Organizations as Systems – Kurosawa, Clausewitz, and Chess

16th Century Market Scene

In order to respond appropriately to the context we find ourselves in, it’s helpful that we be able to correctly define that context. It’s something humans aren’t always good at.

Not too long ago, Sun Tzu’s The Art of War was all the rage as among executives. While the book contains some excellent lessons that have applications beyond the purely military, as someone in my Twitter feed noted recently, “Business is not war”.

[Had I realized that the tweet, in combination with another article, would trigger something in my byzantine thought processes, I would have bookmarked it to give them credit – sorry!]

Business is, indeed, not war. In fact, one of the nuggets of wisdom to be found in Clausewitz’s treatise, On War, is that war is often not war. Specifically, what he is saying is that the reality of a concept often diverges from our (mis)understanding of that concept. Our perception is colored by factors such as our experience, beliefs, and interests. Additionally, our tendency to employ abstraction can be both tool and trap. Ignoring irrelevant detail can simplify reasoning about something, assuming that the detail ignored is actually irrelevant. Ignoring relevant detail can quickly lead to problems.

The game of chess illustrates this. Chess involves strategy and has its origins as an abstract simulation of war. Beyond promoting a very rudimentary type of strategic thought, chess is far from capable of simulating the complex social system of warfare. Perhaps if all the pieces were sentient and had both agency and agenda (bonus points for contradictory ones potentially conflicting with the player’s agenda), it might come closer. Perhaps if the boundaries of the arena were indeterminate, it might come closer. Perhaps if the state of the terrain, the composition and disposition of forces (friend, as well as foe), and the goals of the opponent were less transparent, it might come closer.

In short, the more certainty there is, the less accuracy there is. Where the human aspect is ignored or minimized, you may gain certainty, but it comes at the cost of losing contact with reality. Social systems are highly complex and treating them otherwise is like looking for a gas leak with a lighter – you may be able to do so, but your chances of liking the results are pretty small.

This post was originally planned to be for last week, but I stumbled into a Twitter conversation that illustrates my point (specifically re: leadership and management), so I wrote that first as a preamble. Systems of practice designed for a context where value equals effort expended are unlikely to work well in a knowledge work context where the relationship between effort and value is less direct (where, in fact, the value curve may invert past a certain point). Putting an updated veneer on the technique with data and algorithms won’t improve the results if the technique is fundamentally mismatched to the context (or if there is a disconnect between what you can measure and what you actually want). Sometimes, the most important thing to learn about management is when not to manage.

Disconnects between complex contexts and simplistic practices transcend the management of an organization, reaching into the very architecture of the enterprise itself (both in the organization and its relationship to its ecosystem). Poorly designed organizations (which includes those with no intentional design) can wind up with their employees faced with perverse incentives to act in a manner that conflicts with the best interests of the organization. When the employee is actually under pressure from the organization to sabotage the organization, the problem is not with the employee.

Just as with a software system, social systems have both problem and solution architectures. Likewise, in both cases the quality of the solution architecture is dependent on how well (or not) it addresses the architecture of the problem. Recognizing the various contexts in play and then resolving the conflicts between them (to include resolving challenges arising from the resolution of the original conflicts) is the essence of architectural design, regardless of the type of system (software or social). Rather than a static, one time activity, it is an ongoing need for sensing system health and responding appropriately throughout the lifecycle of the system (in fact, stopping the process will likely hasten the end of the lifecyle by way of achieving a state where the system cannot be corrected).

Advertisement

Storming on Design

From Wikimedia: VORTEX2 field command vehicle with tornado in sight. Wyoming, LaGrange.

My youngest son has recently fallen in love with the idea of being a storm chaser when he gets older. Tweet storms are more my speed. There was an interesting one last week from Sarah Mei regarding the contextual nature of assessing design quality:

Context is a recurring theme for me. While the oldest post with that tag is just under three years old, a search on the term finds hits going all the way back to my second post in October, 2011. Sarah’s tweets resonated with me because in my opinion, ignoring context is a fool’s game.

Both encapsulation and silos are forms of separation of concerns. What differentiates the two is the context that makes the one a good idea and the other a bad idea. Without the context, you can come up with two mutually exclusive “universal” principles.

A key component of architectural design, is navigating the fractals that make up the contexts in which a system exists. Ruth Malan has had this to say regarding the importance of designing “outside the box”:

Russell Ackoff urged that to design a system, it must be seen in the context of the larger system of which it is part. Any system functions in a larger system (various larger systems, for that matter), and the boundaries of the system — its interaction surfaces and the capabilities it offers — are design negotiations. That is, they entail making decisions with trade-off spaces, with implications and consequences for the system and its containing system of systems. We must architect across the boundaries, not just up to the boundaries. If we don’t, we naively accept some conception of the system boundary and the consequent constraints both on the system and on its containing systems (of systems) will become clear later. But by then much of the cast will have set. Relationships and expectations, dependencies and interdependencies will create inertia. Costs of change will be higher, perhaps too high.

This interrelationship can be seen from the diagram taken from the same post:

System Context illustration, Ruth Malan

It’s important to bear in mind that contexts are multi-dimensional. All but the very simplest of systems will likely have multiple types of stakeholders, leading to multiple, potentially conflicting contexts. Accounting for these contexts while defining the problem and while designing a solution appropriate to the problem space is critical to avoiding the high costs Ruth referred to above.

Another takeaway is that context can (and likely will) change over time. Whether it’s changes in terms of staffing (as Sarah noted) or changes in the needs of users or changes in technology, a design that was fit for yesterday’s context can become unfit for today’s and a disaster for tomorrow’s.

Capability Now, Capability Later

Mock tank, British Army in Italy, WWII

In my post “Strategic Tunnel Vision”, I touched on the concept of capability. I discussed how focusing on new capabilities can crowd out existing capabilities and the detrimental effects of that when those existing capabilities are still necessary. I also spoke to how choices about strategic capabilities can trickle down to effect tactical capabilities.

What I failed to do, however, was define what was meant by the term “capability”. That’s a pretty big oversight on my part, because, in my opinion, understanding the concept is critical across all levels of architectural concerns.

Tom Graves, in his “Definitions on capability”, defines the term (along with some related concepts):

— Capability: the ability to do something.

— Capability-based planning: planning to do something, based on capabilities that already exist, and/or that will be added to the existing suite of capabilities; also, identifying the capabilities that would be needed to implement and execute a plan.

— Capability increment: an extension to an existing capability; also, a plan to extend a capability.

— Capability map: a visual and/or textual description of (usually) an organisation’s capabilities.

Yes, I do know that those definitions are terribly bland and generic – and they need to be that way. That’s the whole point: they need to be generic enough to be valid and usable at every possible level and in every possible context – otherwise we’ll introduce yet more confusion to something that’s often way too confused already.

That last paragraph is critical. The concept of “capability” is a high-level one that is useful across multiple levels of architectural concern (ie. application, solution, enterprise IT, and the enterprise itself). Quoting Tom again:

Note what else is intentionally not in that definition of ‘capability‘:

  • there’s no actual doing – it’s just an ability to do something, not the usage of that ability
  • there’s no ‘how’ – we don’t assume anything about how that capability works, or what it’s made up of
  • there’s no ‘why‘ – we don’t assume any particular purpose
  • there’s no ‘who‘ – we don’t assume anything about who’s responsible for this capability, or where it sits in an organisational hierarchy or suchlike

We do need all of those items, of course, as we start to flesh out the details of how the capabilities would be implemented and used in real-world practice. But in the core-definition itself, we very carefully don’t – they must not be included in the definition itself.

The reason why we have to be so careful and pedantic about this is because the relationship between service, capability, function and the rest is inherently recursive and fractal: each of them contains all of the others, which in turn each contain all of the others, and so on almost to infinity. If we don’t use deliberately-generic definitions for all of those items, we get ourselves into a tangle very quickly indeed – as can be seen all too easily in the endless definitional-battles about the relationships between ‘business-function’ versus ‘business-process’ versus ‘business-service’ versus ‘business-capability’ and so on.

In short, it’s a crucial building block in our designs and plans (which is redundant, since design is a form of planning). If we don’t have and can’t get the ability to do something, it’s game over. However, as Tom noted, we need to move beyond the raw ability in order to make effective use of capabilities. We need to think timing and personnel (which will probably largely drive timing anyway). A capability later may well not be as valuable as the same capability right now.

This was brought to mind while skimming a book review on a military strategy site (emphasis added by me):

In March 2015, then-Chief of Staff of the U.S. Army General Raymond T. Odierno admitted to the British newspaper The Telegraph that the so-called special relationship between the United States and Great Britain isn’t what it used to be. “In the past we would have a British Army division working alongside an American army division,” he said, but he feared that in the future British battalions and brigades would have to operate “inside” American units. “What has changed,” Odierno declared, “is the level of capability.”

Later that week, I asked a senior British general about Odierno’s remarks. He replied, deadpan, that although Odierno’s candor was appreciated, his statement was factually incorrect. “We can still field a division,” the general insisted. “It is just a question of how long it takes us to field one.” Potential tanks, he seemed to think, were just as relevant as an actual ones.

The highlighted portion of the quote illustrates my point. Having the capability to do something immediately and the capability to do that same thing at some point in the future are not equivalent (just to be fair to the British Army, the US Army was in this same position during Operation Desert Shield – the initial ground forces that could be deployed were extremely thin). Treating them as equivalent potentially risks disaster.

It should be noted, however, that level of concern will color the perception of the value of a future capability versus a current one. At the tactical level, in business as well as in war, “…first with the most…” is likely a winning move. At the strategic level, however, where resources must be budgeted across multiple initiatives, priorities should dictate which capabilities get preference. Tactical leaders may have to be satisfied with “on time with just enough”.

Regardless of level, a clear assessment of capabilities, what’s available when, is key to making effective decisions.

Pragmatic Application Architecture

I saw a tweet on Friday about a SlideShare deck that looked interesting, so I bookmarked it to read later. As I was reading it this morning, I found myself agreeing with the points being made. When I got to the next to the last slide, I found myself (or at least, this blog) listed alongside some very distinguished company under “Reading Material”.

Thanks, Bart Blommaerts and nice job!

Form Follows Function on SPaMCast 381

SPaMCAST logo

This week’s episode of Tom Cagley’s Software Process and Measurement (SPaMCast) podcast, number 381, features Tom’s essay on Agile adoption, Kim Pries talking about technology’s gender gap, and a Form Follows Function installment on the fallacy of greenfield development.

Tom and I discuss my post “The Seductive Myth of Greenfield Development”. We talk about how nothing’s ever from scratch; even if the system is completely new, those developing it and those using it bring their own baggage to the table.

You can find all my SPaMCast episodes using under the SPAMCast Appearances category on this blog. Enjoy!

Form Follows Function on SPaMCast 377

SPaMCAST logo

This week’s episode of Tom Cagley’s Software Process and Measurement (SPaMCast) podcast, number 377, features Tom’s essay on empathy, Kim Pries talking about the application of David Allen’s concepts for Getting Things Done, and the first Form Follows Function installment for 2016 on organizations and innovation.

Tom and I discuss my post “Changing Organizations Without Changing People”, talking about the need to work with, not against, human nature in the design and operation of organizations.

You can find all my SPaMCast episodes using under the SPAMCast Appearances category on this blog. Enjoy!

The Seductive Myth of Greenfield Development

Greger Wikstrand‘s tweet from earlier this week packed a wealth of inspiration into one image:

The second statement particularly resonated with me: “The present is built on the past.”

How often do we, or those around us, long for a chance to do things “from scratch”. The idea being, without the constraints of “legacy” code, we could do things “right”. While it’s a nice idea, it has no basis in reality.

Rewrites, of course, will involve dealing with existing data. I’ve yet to encounter a system where no one was interested in the data when it was replaced. I’ve shut down a few where there was no interest, but that’s a different story. The need for that existing data will serve as a potent influence on what can or cannot be done with the replacement system. Likewise, its structure. It’s not reasonable to assume that the data will be any less “legacy” than the code.

We might be tempted to believe that brand new systems escape this pitfall. In doing so, we fail to consider that new systems still must deal with the wants, needs, and attitudes of their stakeholders. People, processes, and organization form the ecosystem that new systems must fit into as surely as replacement systems must.

A crucial part of problem solving is having an adequate understanding of the problem. Everything has a backstory. Understanding the backstory is dependent on understanding the ecosystem the thing fits into. This what Sullivan was talking about when he said “…form ever follows function”.

Nothing’s Ex Nihilo.

Technical Debt and Rolling Re-writes (Who Needs Architects?)

If you think building a system is challenging, try maintaining one.

Tom Cagley‘s recent post “Plan to Throw One Away Re-Read Saturday: The Mythical Man-Month, Part 11”, was a good reminder that while “technical debt” may be something currently on the radar for many, it’s far from a new phenomenon. The concept of instant legacy applications was in place when forty years ago when Frederick Brooks wrote his masterpiece, even if they weren’t called that. As Tom observed in the post:

Rarely is the first attempt useful to the end consumer, and the usefulness of that first attempt is less in the code than in the feedback it generates. Software development is no different. The initial conceptual design and anticipated technical architecture of a large project rarely stands up to the rigors of the discovery process, and those designs should be learned from and then thrown away.

The faulty assumptions and design flaws accumulate not only from sprint to sprint leading up to the initial release, but also from release to release. In spite of the fact that a product can be so seriously flawed, throwing it away and starting over is easier said than done. While sunk costs cannot be recovered, too sanguine an attitude towards them may not enhance your credibility with the customer. Having to pay for the same thing over and over can make them grumpy.

This sets up a dilemma, one that frequently leads to living with technical debt and attempting to incrementally patch it up. There are limits, however, to the number of band-aids that can be applied. This might make it tempting to propose a rewrite, but as Erik Dietrich stated in “The Myth of the Software Rewrite”:

Sure, they know things now that they didn’t know when they started on this code 3 years ago. But won’t the same thing be true in 3 years? Won’t the developers then be looking at the code and saying, “this is a mess — if only we knew in 2015 what we now know in 2018!” And, beyond that, what makes you think that giving the same group of people the same marching orders won’t result in the same kind of code?

The “big rewrite from scratch because this is a mess” is a losing strategy.

Fortunately, there is an alternative. Quoting Tom Cagley again from the same post as above:

If change is both inevitable and good (within limits), then both systems and organizations (a type of system) need to be engineered to support and facilitate change. Architecturally, techniques such as modularization, object-oriented design and other processes that foster simplification and incremental change create an environment in which change isn’t avoided, but rather encouraged.

While we may laugh at the image of changing a tire while the vehicle is in motion, it is an accurate metaphor. Customers expect flexibility and change on the go; waiting equals lost business. The keys to evolving in place are having an intentionally designed, modular architecture and an understanding of where the weaknesses lie. Both of these are concerns that reside squarely on the architect’s plate.

Modularity not only makes an application more easily maintainable via separation of concerns, but it also embraces change by making components replaceable. This is one of the qualities that has made microservices such a hot topic, although it would be a mistake to think that microservices are the only way (or best way in all cases) to achieve modularity.

Modularity brings benefits beyond the purely technical as well. Rewrites of a fraction of an application are more easily sold than big-bang efforts. Demonstrating forethought (while you can’t predict what the change will be, predicting the need for change is more of a sure thing) demonstrates concern for the customer’s welfare, which should make for a better relationship.

Being able to throw a system away a little at a time allows us to keep the car on the road while it changes and adapts to changing conditions.

“Bumper Sticker Philosophy” on Iasa Global Blog

Yeah!  Wait, what?

YAGNI: You Ain’t Gonna Need It.

Sound bites are great – short, sweet, clear, and simple. Just like real life, right?

Seductive simple certainty is what makes slogans so problematic. Uncertainty and ambiguity occur far more frequently in the real world. Context and nuance add complexity, but not all complexity can be avoided. In fact, removing essential complexity risks misapplication of otherwise valid principles.

See the full post on the Iasa Global Blog (a re-post, originally published here).

Accidental Architecture

Hillside Slum

I’m not sure if it’s ironic or fitting that my very first post on Form Follows Function, “Like it or not, you have an architecture (in fact, you may have several)”, dealt with the concept of accidental architecture. A blog dedicated to software and solution architecture starts off by discussing the fact that architecture exists even in the absence of intentional design? It is, however, a theme that seems to recur.

The latest recurrence was a Twitter exchange with Ruth Malan, in which she stated:

Design is the act and the outcome. We design a system. The system has a design.

This prompted Arnon Rotem-Gal-Oz to observe that architecture need not be intentional and “…even areas you neglect well [sic] have design and then you’d have to deal with its implications”. To this I added “accidental architecture is still architecture – whether it’s good architecture or not is another thing”.

Ruth closed with a reference to a passage by Grady Booch:

Every software-intensive system has an architecture. In some cases that architecture is intentional, while in others it is accidental. Most of the time it is both, born of the consequences of a myriad of design decisions made by its architects and its developers over the lifetime of a system, from its inception through its evolution.

The idea that an architecture can “emerge” out of skillful construction rather than as a result of purposeful design, is trivially true. The “Big Ball of Mud”, an ad hoc arrangement of code that grows organically, remains a popular design pattern (yes, it’s a pattern rather than an anti-pattern – see the Introduction of “Big Ball of Mud” for an explanation of why). What remains in question is how effective is an architecture that largely or even entirely “emerges”.

Even the current architectural style of the day, microservices, can fall prey to the Big Ball of Mud syndrome. A plethora of small service applications developed without a unifying vision of how they will make up a coherent whole can easily turn muddy (if not already born muddy). The tagline of Simon Brown’s “Distributed big balls of mud” sums it up: “If you can’t build a monolith, what makes you think microservices are the answer?”.

Someone building a house using this theory might purchase the finest of building materials and fixtures. They might construct and finish each room with the greatest of care. If, however, the bathroom is built opening into the dining room and kitchen, some might question the design. Software, solution, and even enterprise IT architectures exist as systems of systems. The execution of a system’s components is extremely important, but you cannot ignore the context of the larger ecosystem in which those components will exist.

Too much design up front, architects attempting to make decisions below the level of granularity for which they have sufficient information, is obviously wrong. It’s like attempting to drive while blindfolded using only a GPS. By the same token, jumping in the car and driving without any idea of a destination beyond what’s at the end of your hood is unlikely to be successful either. Finding a workable balance between the two seems to be the optimal solution.

[Shanty Town Image by Otsogey via Wikimedia Commons.]