Inflection Points and the Ingredients of Innovation

One of my hobbies is the study of history. Not the dry, dusty, “…on this date these people did that” type of history, rather I’m fascinated by the story of how real people interacted with each other and the world around them. I’m interested in the brilliance and the stupidity, the master strokes and the blunders, the nobility and the infamy; all of which are often found combined in the same person. I like the type of history that, in explaining the world of yesterday, is helpful in making sense of the world of today.

World War I is one of those large-scale human tragedies that holds lessons for today. Teetering on the edge of two very different times, it mixed the tools of industrial warfare (machine guns, armored vehicles, and air power) with infantry tactics barely different from those of a hundred years previous. Over a little more than four years, old and new collided cataclysmically. It provides a brutally stark picture of the uneven nature of innovation and how that uneven nature can yield both triumph and tragedy.

So, just what does this have to do with information technology, systems development, and software architecture?

Quite a lot, actually, if you look beyond the surface.

Over the last few weeks, I’ve been having a running conversation with Greger Wikstrand on this blog and Twitter, about various aspects of IT. “We Deliver Decisions (Who Needs Architects?)” wove together some observations about situational awareness and confirmation bias into a discussion about the need for and aim of the practice of architectural design. In response, Greger posted a link to a video with him and Woody Zuill on serendipity in software development. That prompted my last post, “Fixing IT – Too Big to Succeed?”, discussing how an embedded IT approach could be used to foster the organizational agility needed to provide IT both effectively and efficiently.

Greger responded with “Serendipity and successful innovation”, making a number of extremely important points. For example, while fortune does favor the prepared, recognition of an opportunity is not enough. Likewise, an organizational structure that doesn’t impede innovation is important, but not impeding innovation is not the same as actively driving it forward. Evaluation of ideas is critical as well, in order to differentiate “…between the true gold of serendipity and the fool’s gold of sheer coincidence”. That evaluation, however, needs to be nimble as well. He closed the post with an example of small-company innovation on the part of a large organization.

That last part was what brought WWI to mind. The first and second World Wars were very different affairs (both horrible, but in different ways). Quite a lot of the technologies that we think of as being characteristic of the second (e.g. warplanes and tanks), originated in the first. Having a technology and discovering how best to employ it are two very different things. Certainly there was some technical evolution over twenty years, but the biggest change was in how they were employed.

In some cases, the opposite problem was in play. The organizational structures used by the combatants had proved their effectiveness for over a hundred years. Mission tactics, AKA auftragstaktic (whereby a commander, instead of detailed instructions, provides a desired outcome to his subordinates who are then responsible for achieving that outcome within broad guidelines), was also well-known at the time, but while radio existed, it was not yet portable enough to link dispersed mobile front line units with headquarters. Instead, that communication was limited to field phones and runners, and tactical communication was limited by the range of the human voice, forcing units to bunch up.

Without all the ingredients (technology, the understanding of how to employ it, and an effective organizational structure) outcomes are likely to be poor. Information must flow both up and down the hierarchy, otherwise decisions are being made blindly and without coordination. The parts must not only perform their function, but must also collaborate together. Organizations that fail to operate effectively across the full stack fail to operate effectively at all.