Form Follows Function on SPaMCast 411

SPaMCAST logo

This week’s episode of Tom Cagley’s Software Process and Measurement (SPaMCast) podcast, number 411, features Tom’s essay on Servant Leadership (which I highly recommened), John Quigley on managing requirements as a part of product management, a Form Follows Function installment based on my post “Organizations as Systems – ‘Uneasy Lies the Head that Wears the Crown'”, and Kim Pries on software craftsmanship.

Tom and I discuss the danger of trying to use simplistic explanations for the interactions that make up complex human systems. No one has the power to force things in a particular direction, rather the direction comes about as a result of the actions and interactions of everyone involved. It might be comforting to believe that there’s one single lever for change, but it’s wrong.

You can find all my SPaMCast episodes using under the SPAMCast Appearances category on this blog. Enjoy!

Organizations as Systems – “Uneasy Lies the Head that Wears the Crown”

Bavarian Crown and Regalia, Royal Treasury Munich

 

One of the benefits of having a (very) wide range of interests is that every so often a flash of insight gets dropped into my lap. In this case, it was a matter of “We must recognise that single events have multiple causes” showing up as a suggested read from Aeon on the same day that Thomas Power retweeted this:

The image in the tweet is an excerpt from an interview with Rory Stewart, Conservative Member of Parliament for Penrith in the UK. The collision of themes between the two articles struck me.

“You get there and you pull the lever, and nothing happens.”

The behavior of a system is determined not by the structure of the components of that system, but by the relationships and interactions between those components. Moreover, those relationships and interactions are dynamic and complex, even when that’s contrary to the designer’s intent. In fact, the gap between the behavior as intended and as experienced introduces a tension. I would argue that it’s less a matter of nothing happening when the “lever” is pulled and more that something different from what’s expected happens. Rather than simple cause and effect, “if this, then that”, multiple factors are in play.

In mechanical systems, parts wear, subtly changing the physics of the mechanism. Foreign objects invading the system can impose change in a more dramatic fashion. Context, both that of the system’s internals and its environment, influences its operation.

As was noted in the Aeon article, agency adds to the complexity. In social systems, all of the “components” are individuals with agency, making those systems chaotic in at least the colloquial sense of the word. Using Tom Graves’ sense-making framework, SCAN, these interactions fall into the more uncertain quadrants, either “Ambiguous but Actionable” or “Not-known, None-of-the-above”. Attempting to deal with them as though they fell into the “Simple and Straightforward” quadrant increases the likelihood of getting unexpected results.

Learning/sense-making is critical to dealing with change, whether internal or external (or both). The manner in which change is appreciated and reacted to, affects the health of the system. Consider three boilers: one where pressure is continuously monitored and adjusted, one which is equipped with a pressure relief valve which will open prior to a catastrophic failure, and one where problems are signaled by an explosion. It’s a trivial exercise to come up with examples of social systems, from businesses all the way up to political systems, using the third method. It’s probably a more interesting exercise to consider why that’s the case for so many.

In a recent post, “Architecting the shadows”, Tom Graves discussed the phenomenon of ad hoc, unofficial “shadow” organizational interactions that arise in order to get work done:

In SCAN terms, we could summarise the generic positioning of all ‘shadow’ functions – shadow-IT, shadow-business-models, shadow-management and more – much as follows:

Scan Diagram: Official vs. Shadow

In other words, the ‘shadow’-world exists to deal with and resolve all the uncertainties and over-simplifications that overly-mechanistic management models tend to overlook. Even in more aware management-models, in which some exploration of the uncertain is officially sanctioned and allowed, the shadow-world will still always need to exist – particularly whenever the work gets closer towards real-time action:

Scan Diagram: Official vs. Shadow showing sanctioned Shadow Activity

In closing the post, Tom makes the following observation:

As the literal ‘the architecture of the enterprise’, a real enterprise-architecture must, by definition, cover every aspect of the enterprise – including all of the ‘shadow’-elements. And yet, also by definition, those ‘shadow’-elements cannot be brought ‘under control’ – not least because they deal with the themes and factors that are beyond the reach of conventional concepts of ‘control’.

The “conventional concepts of ‘control'”, the deluded belief that complex interactions can be managed as though they were simple, poses an immense risks to organizations. Even attempting to treat those interactions as merely complicated, rather than complex, introduces a gap between reality and perception, between “the way we do things” and the way things actually get done. When the concept and reality of the system’s interactions differ, it’s more likely that the components of the system will wind up working at cross-purposes.

In a comment on Tom’s post, I noted that where the shadow elements are a “French Resistance”, flouting the rules in order to actually get work done, that’s a red flag.

The most important thing to learn about management and governance is knowing when and how to manage or govern and more importantly, when not to. Knowing what can actually be controlled is an important first step.

Back to the OODA – Making Design Decisions

OODA Loop Diagram

A few weeks back, in my post “Enterprise Architecture and the Business of IT”, I mentioned that I was finding myself drawn more and more toward Enterprise Architecture (EA) as a discipline, given its impact on my work as a software architect. Rather than a top-down approach, seeking to design an enterprise as a whole, I find myself backing into it from the bottom-up, seeking to ensure that the systems I design fit the context in which they will live. This involves understanding not only the technology, but also how it interacts, or not, with multiple social systems in order to (ideally) carry out the purpose of the enterprise.

Tom Graves is currently in the middle of a series on whole-enterprise architecture on his Tetradian blog. The third post of the series, “Towards a whole-enterprise architecture standard – 3: Method”, focuses on the need for a flexible design method:

But as we move towards whole-enterprise architecture, we need architecture-methods that can self-adapt for any scope, any scale, any level, any domains, any forms of implementation – and that’s a whole new ball-game…

He further states:

the methods we need for the kind of architecture-work we’re facing now and, even more, into the future, will need not only to work well with any scope, any scale and so on, but must have deep support for non-linearity built right into the core – yet in a way that fully supports formal rigour and discipline as well.

To begin answering the question “But where do we start?”, Tom looks at the Plan-Do-Check-Act (PDCA) cycle, which he finds wanting because:

… the reality is that PDCA doesn’t go anything like far enough for what we need. In particular, it doesn’t really touch all that much on the human aspects of change

This is a weakness that I mentioned in “OODA vs PDCA – What’s the Difference?”. PDCA, by starting with “Plan” and without any reference to context, is flawed in my opinion. Even if one argues that assessing the context is implied, leaving it to an implication fails to give it the prominence it deserves. In his post, Tom refers to ‘the squiggle’, a visualization of the design process:

Damien Newman's squiggle diagram

In an environment of uncertainty (which pretty much includes anything humans are even peripherally involved with), exploration of the context space will be required to understand the gross architecture of the problem. In reconciling the multiple contexts involved, challenges will emerge and will need to be integrated into the architecture of the solution as well. This fractal and iterative exploratory process is well represented by ‘the squiggle’ and, in my opinion, well described by the OODA (Observe, Orient, Decide, and Act) loop.

In “Architecture and OODA Loops – Fast is not Enough”, I discussed how the OODA loops maps to this kind of messy, multi-level process of sense-making and decision-making. The “and” is extremely important in that while decision-making with little or no sense-making may be quick, it’s less likely to effective due to the disconnect from reality. On the other hand, filtering and prioritizing (parts of the Orient component of the loop) is also needed to prevent analysis paralysis.

In my opinion, its recognition and handling of the tension between informed decision-making and quick decision-making makes OODA an excellent candidate for a design meta-method. It is heavily subjective, relying on context to guide appropriate response. That being said, an objective method that’s divorced from context imposes a false sense of simplicity with no real benefit (and very real risks).

Reality is messy; our design methodology should work with, not against that.

[OODA Loop diagram by Patrick Edwin Moran via Wikimedia Commons]

Form Follows Function on SPaMCast 389

SPaMCAST logo

This week’s episode of Tom Cagley’s Software Process and Measurement (SPaMCast) podcast, number 389, features Tom’s essay on Agile acceptance testing, Kim Pries talking about soft skills, and a Form Follows Function installment on sense-making and decision-making in the practice of software architecture.

Tom and I discuss my post “OODA vs PDCA – What’s the Difference?”. We talk about what differentiates John Boyd’s Observe-Orient-Decide-Act loop from the Plan-Do-Check-Act cycle made famous by Deming.

You can find all my SPaMCast episodes using under the SPAMCast Appearances category on this blog. Enjoy!

Form Follows Function on SPaMCast 385

SPaMCAST logo

This week’s episode of Tom Cagley’s Software Process and Measurement (SPaMCast) podcast, number 385, features Tom’s essay on Agile portfolio metrics, Kim Pries talking about the value of diversity, and a Form Follows Function installment on sense-making and decision-making in the practice of software architecture.

Tom and I discuss my post “Architecture and OODA Loops – Fast is not Enough”. We talk about how making good decisions is the very essence of the practice of software architecture. I relate John Boyd’s Observe-Orient-Decide-Act loop to designing software systems.

You can find all my SPaMCast episodes using under the SPAMCast Appearances category on this blog. Enjoy!

OODA vs PDCA – What’s the Difference?

PDCA Loop

In my post “Architecture and OODA Loops – Fast is not Enough”, I stated that sense-making and decision-making were critical skills for the practice of software architecture. I further stated that I found the theories of John Boyd, particularly his OODA loop, useful in understanding and describing effective sense-making and decision-making. My conclusion was that in order to decide and act in the most effective manner possible, one must observe the context as effectively as possible and orient, or make sense of those observations, as effectively as possible. In other words, the quality of decision depended on the quality of the cognition.

One of the many things that I enjoy about blogging and engaging on media like Twitter and LinkedIn is the feedback. The questions and comments I get in response to one post ensure that I don’t have to worry about writer’s block getting in the way of my next one. In this instance, Greger Wikstrand obliged, providing the topic for this post:

As I replied to Greger, I was familiar with the PDCA cycle and it has some similarities to Boyd’s OODA loop, but I preferred OODA for a variety of reasons. Before I get into those reasons, however, it would be useful to lay out what distinguishes the two methods. In a guest post on the blog Slightly East of New, “PDCA vs. OODA — Why not take both?”, Deane Lenane does just this:

Some people who are familiar with the canon of both of these men’s work, often fall into the error of seeing the O-O-D-A loop as a function of the P-D-C-A loop or vice versa. I think this is a mistake.

The P-D-C-A cycle or loop is primarily an analytical approach that can be used with great success in a completely internal manner. One does not need to consult the external environment or adjust to unfolding circumstances to make the P-D-C-A loop work. P-D-C-A can be used with great success on the shop floor with the data that is available. Analysis which involves the use of a more or less complete data set to reach a conclusion. We use the data to make a decision about how to proceed, we than check and act to confirm or reject the hypothesis that our analysis has led us to.

O-O-D-A is more concerned with synthesizing an action out of an incomplete data set. Since we can never recognize all of the variables that we are forced to deal with in any environment, we must be able to make a decision that we believe will give us the highest probability for success. The synthesis of an action from the observation and orientation of a complex and mysterious environment, subject to frequent and unpredictable change, is the essence of the O-O-D-A loop.

My conclusion is that P-D-C-A is primarily involved with analysis perhaps using some synthesis and that O-O-D-A is primarily involved with synthesis using all of the analytical data points possible but considering that the data set will always be largely incomplete.

Three aspects of Lenane’s differentiation point me toward OODA: “environment” (i.e. the evaluation of the external context we must deal with rather than the more inward looking PDCA), “synthesizing an action”, and “incomplete data set”. Systems exist in an environment, an ecosystem with a back-story. Failing to account for those ensures problems if not outright failure. Likewise, our aim is to make a decision to the best of our ability under uncertainty. These concepts mesh nicely with the practice of architecture in my opinion.

Another aspect relates back to the PCA acronym in Greger’s tweet. That took some research for us to find a good resource, because:

However, Greger did find one (slide 3 of this deck). Essentially, it states that Perception of the world leads to Cognition which leads to Action that changes the world. Perception being the key word here. How we perceive reality is arguably more important than reality itself, because that perception will color our thinking that drives our action. The Orient portion of the OODA explicitly recognizes that our observations are filtered through our experience, education, biases, etc. Understanding and adjusting for this (to the extent we can) is important. As Tom Graves observed in “Enterprise-architect – applied-scientist, or alchemist?”:

Perhaps the most important thing here is to notice things that don’t fit our expectations – they’re often very easy to miss, especially given Gooch’s Paradox, that “things not only have to be seen to believed, but also have to be believed to be seen”.

Software architecture involves making decisions in the presence of uncertainty. In order to make the best decisions possible, we need to have the best possible grasp of our context (n.b. remembering that we are part of that context). We also need to remember that the context is not static. Each action (or inaction, for that matter) can lead to the emergence of some new issue. We can’t operate with a “one and done” philosophy.

[PDCA Loop diagram by Tagimaguiter via Wikimedia Commons]

Architecture and OODA Loops – Fast is not Enough

OODA Loop Diagram

Sense-making and decision-making are critical skills for the practice of software architecture. Creating effective solutions (i.e. the collection of design decisions that make up the product) is dependent on understanding the architecture of the problem. In other words, the quality of our decisions depends on the quality of our understanding of the context those decisions were intended to address. As Paul Homer recently observed in his post “Thinking Problems”:

Thinking well is a prerequisite for handling complex problems, with or without technology. If people can’t clearly understand the issues, then any attempt to solve them is as equally likely to just make them worse. If they break down the problem into a convoluted mess of ill fitting parts, then any solutions comprised of these is unlikely to work effectively, and is more likely to make things worse. Software, as it is applied to sophisticated problems, is inordinately complex and as such can not be constructed without first arriving at a strong understanding of the underlying context. This comes from observations, but it doesn’t come together until enough thinking has been applied to it to get it into a usable state.

Systems, both automated and social (particularly social), tend to display complex, emergent behavior. The ecosystem for automated systems is a social system that may comprise other social systems. The development and operation of automated systems is, in itself, a social system. Agility in sense-making, decision-making, and execution is critical to effective operation in this environment.

This need for agility is why, when it comes to thinking about thinking, one of the theorists whose ideas I pay particular attention to is John Boyd. Boyd, a US Air Force fighter pilot, gained fame for his theories of cognition. Although initially developed in relation to the tactics of aerial combat, they were found to have a wider application (applying to strategy as well and not being limited to military domains). At its core, the theory was that to defeat an opponent, one must get inside their OODA (Observe, Orient, Decide, and Act) loop. In other words, the person that could observe circumstances, make sense of them (orient), decide on a response and then act on that decision quickest would have the advantage.

Simple explanations of Boyd’s theories are problematic. As Tom Graves points out in his post “Seven sins, sensemaking and OODA”:

…the OODA ‘loop’ isn’t a linear loop – it’s fractal, recursive, re-entrant, looping through itself at multiple levels all at the same time. There’s sensemaking within action, action within sensemaking, decisions affecting how we sense and observe. It’s fractal.

My use of the word “quickest” earlier in the post regarding acting on a decision poses a problem. As I noted in the title, fast is not enough. One could gain speed by just acting without any observation, orientation, or decision. In my opinion, however, the result would most likely be flailing blindly. Too little analysis is detrimental.

Likewise, though, trying to deal with too much information poses a problem. From a conversation with Greger Wikstrand:

Greger rightly observes that the information available is out of our control, but our decision to filter or process it is under our control. By intentionally and intelligently filtering, we can traverse the loop quicker and more effectively.

Whether engaged in application architecture, solution architecture, or enterprise architecture, sense-making and decision-making are crucial components of the practice. That sense-making and decision-making is a continual process throughout the lifetime of the system, not just an initial or occasional phase. Constant understanding of the context the system exists within is vital because:

[OODA Loop diagram by Patrick Edwin Moran via Wikimedia Commons]