You can’t always get what you want…

Lucifer

You can’t always get what you want
But if you try sometimes well you just might find
You get what you need

When it comes to systems, you can’t always get what you want, but you do get what you design (intentionally or not), whether it’s what you need or not.

In other words, the architecture of systems, both social and software, evolve through some combination of intentional design and accidental emergence. Regardless of which end of the continuum the system leans toward, the end result will reflect the decisions made (or not made) in relation to various stimuli. Regarding businesses (a social system), Ruth Malan, in her February 2012 “Trace in Sand” post, put it this way:

I have been talking about agility in terms of evolutionary ecology, but with the explicit recognition that companies, comprised after all of individuals, attempt to speed and alter and intervene and interject and intercede and (I’m looking for the right word here) shape evolutionary processes with intentional actions — concerted, but also emergent from more and less choreographed, actions and intentions. Being bumped along by the unpredictable interactions of others, some from within, but also from “invasive species” from other ecosystems looking for new applications for their adaptable, mutable capability set.

Organizations create and participate in business ecologies. They build up the relationships that stabilize parts of the broader ecosystem, and create conditions for organizational forms to thrive there. They create products, they create the seeds of the next generation of harvest. They produce variants on their family tree, to target and develop niches.

Ruth further notes that while business adapt and improve in some cases, in other case they have “…become too closely adapted to and integrated within an ecosystem that has been replaced or significantly restructured by some landscape reshaping change…”. People generally refer to this phenomenon as disruption and the way they refer to it would seem to imply that it’s something that happens to or is done to a company. The role of the organization in its own difficulties (or demise) isn’t, in my opinion, well understood.

Last Friday, I saw a tweet from Noah Sussman that provides a useful heuristic for predicting the behavior of any large social system:

the actual reason behind the behavior:

It’s not that people are actively working to harm the organization, but that when there is no leadership, where there is no design, where there is no learning, the system ossifies and breaks down. Being perfectly adapted to an ecosystem that no longer exists is indistinguishable from being poorly adapted to the present context. I’m reminded of what Tom Graves stated in “The game of enterprise-architecture”: “things work better when they work together, on purpose”.

Without direction, entropy emerges where coherence is needed.

This is not to say, however, that micro-management is the answer. Too much design/control is as toxic as too little. This is particularly the case when the management system isn’t intentionally designed. Management that is both ad hoc and rigid can cause new problems while trying to solve existing ones. This is illustrated by another tweet from Friday:

The desire to avoid the “…embarrassment of cancellation” led to the decision to risk the lives of a plane load of “…foreign TV and radio journalists and also other foreign notables…”. I suspect that the fiery deaths of those individuals would have been an even bigger embarrassment. The system, however, led to the person who had the decision-making power to take that gamble.

The system works the way you built it, even when you didn’t intend to build it to work that way.

Full Stack Enterprises (Who Needs Architects?)

In my last post, “Locking Down the Prisoners: Control, Conflict and Compliance for Organizations”, I returned to a topic that I’ve been touching on periodically over the last year, organizations as systems, which overlaps significantly with the topic of enterprise architecture (not to be confused with enterprise IT architecture of which EA is a superset). While I do not pretend to be an expert on the subject, the fractal nature of the environment I work within as a software architect makes it difficult (perhaps even dangerous) to ignore.

Systems reside within ecosystems, which being systems themselves, reside within an ecosystem of their own. Both systems and ecosystems are mutually influencing and that influence must be understood to the extent possible and accounted for, both upstream and down. Where this fails to happen, coherence between information systems and social systems breaks down. Significantly, issues in systems at higher levels of granularity can negate positive aspects of the systems that comprise them.

This is probably a good place to point out that my use of terms like “social system” is not meant to remove the human aspect. On the contrary, social systems are intensely human in nature. Where those systems are dysfunctional, it is individuals that ultimately pay the price. The intent, is to point out the inter-relationships that make up our environment.

Consider the hypothetical systems outlined in “Making and Taming Monoliths”. Assuming all of the systems involved were modular enough and technically capable of interoperation, it would not be enough. Considerations of data architecture (starting with, which system is authoritative?) could spike the effort, or at least seriously delay it. Organizational structure (aka Conway’s Law), behavior (management, governance, process), and psychology/culture could all either impel or impede. A perfect example of this concept is the near requirement that DevOps be in place where microservices are used – it may be possible to develop and deploy microservices in other process models, but it will involve far more difficulty.

“Alignment” is a term that is often mentioned in terms of IT. However, even if an organization’s IT systems are perfectly aligned with the business units they support, it will do little good if those units are working at cross purposes. Just as the full stack of an application requires coherent, intentional design to work well, so too does an enterprise. This does not, however, imply that a rigid, top-down mode of operation is needed; it’s actually quite the opposite.

In “Auftragstaktik and fingerspitzengefühl”, Tom Graves described the UK’s World War II air defense system as an example of an effective enterprise. The two German words refer to the concepts of situational awareness (fingerspitzengefühl) and mission tactics (Auftragstaktik), which underlay the success of the system. In essence, situational awareness at each level was enabled by the flow of information up and down the hierarchy, while decisions appropriate to each level were made at that level informed by understanding of the situation and the commander’s intent. This is not just a matter of process. As Tom noted:

Another key element is organisational-culture – whether the culture invites or dissuades individual judgement within real-time action (auftragstaktik), elicitation and capture of real-world subtleties (for fingerspitzengefühl) and/or whistleblower-type algedonic responses.

An organization must not only be structured so as to achieve its aims, the drive to do so must be part of its DNA. Even when some components of the organization may have remits that conflict with others (think audit, accounting, InfoSec, etc.), the ultimate direction of all parts should harmonize. Structuring a system so as to resolve conflicts is an architectural practice, regardless of the type of system.

Locking Down the Prisoners: Control, Conflict and Compliance for Organizations

Newgate Prison Inmates

The most important thing to learn about management and governance is knowing when and how to manage or govern and more importantly, when not to.

The story is told about a very new and modern penal facility, the very epitome of security and control. Each night, precisely at 11:00 PM, the televisions were shut off and the inmates were herded into their cells for lights out. Since the inmates tended to dislike their enforced bedtime, fights would ensue during the lockdown and throughout the night when the cells needed to be opened (both for purposes of head counts and to respond to the inevitable conflicts caused by locking people in close quarters). If the problems were pervasive enough, an entire housing unit might be punished by – wait for it – being confined to their cells (perpetuating the cycle).

Management of the facility was at a loss on what to do. The conflict was causing disruption in the day-to-day activities. This disruption further exacerbated tensions. The fights led to injuries to both staff and inmates, raising costs and risk of civil litigation, as well as causing staffing problems.

The answer was simple – stop the lockdowns. When the policy was reviewed objectively, it was obvious that enforcement was yielding no benefits to offset the many costs. In fact, stopping enforcement actually increased security by reducing tensions and causing the night owls to sleep in during the day. In a real-life zen moment, it was realized that letting go of the illusion of control provided real control (or at least something closer to it).

Most organizations could benefit from a similar epiphany.

This is not to suggest that process, management, and governance are unnecessary, far from it. Instead, it’s important that the system by which things are run is…systemic. As Tom Graves likes to say, “…things work better when they work together, on purpose”. Intentional design applies to social systems, just as it applies to software systems. Ad hoc evolution, by way of disjointed decisions unencumbered with any coherence, lead to accidental structures. Entropy emerges.

This can be seen in a tweet from Charles T. Betz:

Or, as Gary Hamel tweeted:

The alternative is to do as Yves Morieux stated in his TED talk: “We need to create organizations in which it becomes individually useful for people to cooperate.” This involves a ruthless attention to cause and effect. This involves creating environments where unnecessary friction is removed and necessary friction is understood to be necessary by all involved. It’s a lot easier to get compliance when it’s easier to comply and a lot easier to get conflict when you provoke it.

Innovation, Agility, and the Big Ball of Mud in Meatspace

French infantry in a trench, Verdun 1916

Although the main focus of my blog is application and solution architecture, I sometimes write about process and management issues as well. Conway’s law dictates that the organizational environment strongly influences software systems. While talking with a colleague recently, I stated that I see organizations as systems – social systems operating on “hardware” that’s more complex and less predictable than that which hosts software systems (i.e. people, hence the use of “Meatspace” in the title). The entangling of social and software systems means that we should be aware of the architecture of the enterprise at the least in so far as it will affect the IT architecture of the enterprise.

Innovation and agility are hot topics. Large corporations, by virtue of their very size are at a disadvantage in this respect. In a recent article for Harvard Business Review, “The Core Incompetencies of the Corporation”, Gary Hamel discussed this issue. Describing corporations as “inertial”, “incremental” and “insipid”, he notes that “As the winds of creative destruction continue to strengthen, these infirmities will become even more debilitating”.

The wonderful thing (at least in my mind) about Twitter is that it makes it very easy for two people in the UK and two people in the US to hold an impromptu discussion of enterprise architecture in general and leadership and management issues in particular (on a Saturday morning, no less). Dan Cresswell started the ball rolling with a quote from a Hamel’s article: “most leaders still over-value alignment and conformance and under-value heterodoxy and heresy”. Tom Graves replied that he would suggest that “…heresy is a _necessary_ element of ‘working together’…”. My contribution was that I suspect that “together” is part of the problem; they don’t know how to integrate rebels and followers, therefore the heretics are relegated to a skunkworks or given the sack. Ruth Malan cautioned about limits, noting that “A perpetual devil’s advocate can hold team in perpetual churn; that judgment thing…by which I simply mean, sometimes dissent can be pugnacious, sometimes respectful and sometimes playful; so it depends”.

Ruth’s points re: “…that judgment thing…” and “…it depends” are, in my opinion, extremely important to understanding the issue. I noted that “that judgment thing” was a critical part of management and leadership. This is not in the sense that managers and leaders should only be the ones to exercise judgment, but that they should use their judgment to integrate, rather than eliminate, the heresies so that the organization does not stagnate. There is a need for a “predator”, someone to challenge assumptions, in the management realm as much as there is a need for one in the design and development realm. Likewise, an understanding of “it depends” is key. Neither software systems nor social systems are created and maintained via following a recipe.

While management practices are part of the problem, it’s naive to concentrate on that to the exclusion of all else. Tom Graves is fond of saying, “Things work better when they work together, on purpose”. This is a fundamental point. As he observed in “Dotting the joins (the JEA version)”:

Every enterprise is a system – an ‘ecosystem with purpose’ – constrained mainly by its core vision, values and other drivers. Within that system, everything ultimately connects with everything else, and depends on everything else: if it’s in the system, it’s part of the system, and, by definition, the system can’t operate without it.

The system must be structured to manage, not ignore complexity. Without an intentional design, things fall through the cracks. Tom again, from the same post:

To do something – to do anything, really – we need to know enough to get it to work right down in the detail of real-world practice. When there’s a lot of detail to learn, or a lot of complexity, we specialise: we choose one part of the problem, one part of the context, and concentrate on that. We get better at doing that one thing; and then better; and better again. And everyone can be a specialist in something – hence, given enough specialists, it seems that between us we should be able to do anything. In that sense, specialisation seems to be the way to get things done – the right way, the only way.

Yet there’s a catch. What specialisation really does is that it clusters all of its attention in one small area, and all but ignores the rest as Somebody Else’s Problem. It makes a dot, somewhere within what was previously a joined-up whole. And then someone else makes their own dot, and someone else carves out a space to claim to make their dot. But there’s nothing to link those dots together, to link between the dots – that’s the problem here.

Hamel’s use of the word “incremental” points the way to diagnosing the problem – enterprises have grown organically, rather than springing to life fully formed. Like a software system that has grown by sticking on bits and pieces without refactoring, social systems can become an example of Foote and Yoder’s “Big Ball of Mud” as well. Uncoordinated changes made without considering the larger system leads to a sclerotic mess, regardless of whether the system in question is social or software. My very first post on this blog, “Like it or not, you have an architecture (in fact, you may have several)”, sums it up. The question, is whether that architecture is intentional or not.