Inflection Points and the Ingredients of Innovation

One of my hobbies is the study of history. Not the dry, dusty, “…on this date these people did that” type of history, rather I’m fascinated by the story of how real people interacted with each other and the world around them. I’m interested in the brilliance and the stupidity, the master strokes and the blunders, the nobility and the infamy; all of which are often found combined in the same person. I like the type of history that, in explaining the world of yesterday, is helpful in making sense of the world of today.

World War I is one of those large-scale human tragedies that holds lessons for today. Teetering on the edge of two very different times, it mixed the tools of industrial warfare (machine guns, armored vehicles, and air power) with infantry tactics barely different from those of a hundred years previous. Over a little more than four years, old and new collided cataclysmically. It provides a brutally stark picture of the uneven nature of innovation and how that uneven nature can yield both triumph and tragedy.

So, just what does this have to do with information technology, systems development, and software architecture?

Quite a lot, actually, if you look beyond the surface.

Over the last few weeks, I’ve been having a running conversation with Greger Wikstrand on this blog and Twitter, about various aspects of IT. “We Deliver Decisions (Who Needs Architects?)” wove together some observations about situational awareness and confirmation bias into a discussion about the need for and aim of the practice of architectural design. In response, Greger posted a link to a video with him and Woody Zuill on serendipity in software development. That prompted my last post, “Fixing IT – Too Big to Succeed?”, discussing how an embedded IT approach could be used to foster the organizational agility needed to provide IT both effectively and efficiently.

Greger responded with “Serendipity and successful innovation”, making a number of extremely important points. For example, while fortune does favor the prepared, recognition of an opportunity is not enough. Likewise, an organizational structure that doesn’t impede innovation is important, but not impeding innovation is not the same as actively driving it forward. Evaluation of ideas is critical as well, in order to differentiate “…between the true gold of serendipity and the fool’s gold of sheer coincidence”. That evaluation, however, needs to be nimble as well. He closed the post with an example of small-company innovation on the part of a large organization.

That last part was what brought WWI to mind. The first and second World Wars were very different affairs (both horrible, but in different ways). Quite a lot of the technologies that we think of as being characteristic of the second (e.g. warplanes and tanks), originated in the first. Having a technology and discovering how best to employ it are two very different things. Certainly there was some technical evolution over twenty years, but the biggest change was in how they were employed.

In some cases, the opposite problem was in play. The organizational structures used by the combatants had proved their effectiveness for over a hundred years. Mission tactics, AKA auftragstaktic (whereby a commander, instead of detailed instructions, provides a desired outcome to his subordinates who are then responsible for achieving that outcome within broad guidelines), was also well-known at the time, but while radio existed, it was not yet portable enough to link dispersed mobile front line units with headquarters. Instead, that communication was limited to field phones and runners, and tactical communication was limited by the range of the human voice, forcing units to bunch up.

Without all the ingredients (technology, the understanding of how to employ it, and an effective organizational structure) outcomes are likely to be poor. Information must flow both up and down the hierarchy, otherwise decisions are being made blindly and without coordination. The parts must not only perform their function, but must also collaborate together. Organizations that fail to operate effectively across the full stack fail to operate effectively at all.

Fixing IT – Too Big to Succeed?

Continuing our discussion that I mentioned in my last post, Greger Wikstrand tweeted the following:

BTW @GeneHughson, I think you'll enjoy our video on #serendipity https://t.co/zTlKIDp8kh with me and @woodyzuill

— Greger Wikstrand PhD (@GregerWikstrand) November 11, 2015

I encourage you to watch the video, it’s short (7:39) and makes some important points, which I’ll touch on below.

Serendipity, when it occurs, is a beautiful thing. Serendipity can occur when heads-down order taking is replaced with collaboration. Awareness of business concerns on the part of IT staff enhances their ability to work together with end users to create effective solutions. Awareness, however, is insufficient.

Woody Zuill‘s early remarks about the bureaucracy that gets in the way of that serendipity struck a nerve. Awareness of how to address a need does little good if the nature of the organization either prevents the need from being addressed. Equally bad is when a modest need must be bloated beyond recognition in order to become “significant” enough for IT intervention. Neither situation works well for the end users, IT, or the business as a whole.

So what’s needed to remedy this?

In “Project lead time”, John Cook asserted that project lead team was related to company size:

A plausible guess is that project lead time would be proportional to the logarithm of the company size. If a company with n employees has a hierarchy with every manager having m subordinates, the number of management layers would be around logm(n). If every project has to be approved by every layer of management, lead time should be logarithmic in the company size. This implies huge companies only take a little longer to start projects than medium-sized companies, and that doesn’t match my experience.

While I agree in principle, I suspect that the dominant factor is not raw size, but the number of management layers Cook refers to. In my experience, equally sized organizations can be more or less nimble depending on the number of approvals and the reporting structure of the approvers. Centralized, shared resources tend to move slower than those that are federated and dedicated. Ownership increases both customer satisfaction and their sense of responsibility for outcomes. Thus, embedding IT staff in business units as integral members of the team seems a better choice than attempting to “align” a shifting set of workers temporarily assigned to a particular project.

The nature of project work, when combined with a traditional shared resource organization, stands as a stumbling block to effective, customer-centric IT. IT’s customers (whether internal, external, or both) are interested in products, not projects. For them, the product isn’t “done” until they’re no longer using it. While I won’t go so far as to say #NoProjects, I share many of the same opinions. Using projects and project management to evolve a product is one thing, I’ve found project-centric IT to be detrimental to all involved.

Having previously led embedded teams (which worked as pseudo-contractors: employed by IT, with the goals and gold coming from the business unit), I can provide at least anecdotal support to the idea that bringing IT and its customers closer pays dividends. Where IT plays general contractor, providing some services and sub-contracting other under the direction of those paying the bills, IT can put its expertise to use meeting needs by adding value rather than just acting as order-takers. This not only allows for the small wins that Woody Zuill mentioned, but also for integrating those small wins into the overall operation of the business units, reducing the number of disconnected data islands.

Obviously there are some constraints around this strategy. Business units need to be able to support their IT costs out of their own budget (which sounds more like a feature than a bug). Likewise, some aspects of IT are more global than local. The answer there, is matching the scope of the IT provider unit with the scope of the customer unit’s responsibility. Federating and embedding allows decisions to be made closer to the point of impact by those with the best visibility into the costs and benefits. Centralization forces everything upwards and away from those points.

Operating as a monolithic parallel organization that needs to be “aligned” to the rest of the organization sounds like a system that’s neither effective nor efficient, certainly not both. When the subject is the distance between the decision and its outcome, bigger isn’t better.