Form Follows Function on SPaMCast 471

SPaMCAST logo

It’s time for another appearance on Tom Cagley’s Software Process and Measurement (SPaMCast) podcast.

Last week’s episode, number 471, features Tom’s essay on the top 20 transformation killers. Jeremy Berriault‘s QA corner is about involving testers in the requirements process. My Form Follows Function segment rounds out the podcast, covering my post “Systems Thinking Complicates Things”.

In this installment, Tom and I talk about how simplistic analysis is unlikely to fit a complex problem. We illustrate this by talking about the game of “rock, paper, scissors” (then graduate to “rock, paper, scissors, lizard, Spock”!). I don’t really think that’s what’s meant by game theory, but hey, it’s fun (and illustrative) nonetheless.

You can find all the SPaMCast episodes I’m in under the SPaMCast Appearances category on this blog. Enjoy!

Advertisement

Form Follows Function on SPaMCast 467

SPaMCAST logo

It’s time for another appearance on Tom Cagley’s Software Process and Measurement (SPaMCast) podcast.

This week’s episode, number 467, features Tom’s excellent essay on value (value is one of those simple-seeming, but complex concepts). Jeremy Berriault‘s QA corner covers testing in difficult circumstances. I bat cleanup with a Form Follows Function segment discussing my post “Management, Simple and Wrong – Semantics, Systems, and Self-Correction”.

In this installment, Tom and I talk about a post that came about, not intentionally, but because I got into a conversation on Twitter that played out along the lines of the post I really intended to write (dealing with organizations as complex, multi-level systems). When life hands you your topic, it’s generally a good idea to chase it to the conclusion!

You can find all the SPaMCast episodes I’m in under the SPaMCast Appearances category on this blog. Enjoy!

Systems Thinking Complicates Things

4th UK Rock Paper Scissors Championships by James Bamber via Wikimedia

 

I’ve had the honor and pleasure of appearing as a regular on Tom Cagley‘s SPaMCast podcast for almost three years now. Before I write one of my “Form Follows Function on SPaMCast x” posts, I always listen to the podcast to make sure that the summary is right (the implication being, relying purely on my memory won’t be right). I got a bonus while writing up last week’s appearance, because Tom asked an excellent question that deserved its own post: does thinking about a problem (legacy systems, in the instance of last week’s discussion) holistically/systematically complicate things?

Abso-freakin’-lutely.

It is much easier to avoid all the twists and turns and possibilities inherent in systems thinking. A simpler approach, picking one lever to pull/one button to push, makes it much easier to come up with a solution.

It just doesn’t work very well at coming up with solutions that actually work.

When there is a mismatch in complexity between problem and solution architectures, this mismatch will be an additional problem to deal with. This will apply when the solution is more complex than the problem space warrants and when the opposite is the case. Solutions that fail to account for the context they will encounter are vulnerable. This is the idea behind the quote attributed to Albert Einstein: “Everything should be made as simple as possible, but not simpler.”

Human nature can push us to fix problems quickly, and quick will generally equate to simple. It takes time to analyse the angles and consider the alternatives. How often have you seen people ask for “the best” way to do something absent any context? How often have you seen people ask “why would someone ever do that?”

I’ll answer that by asking 3 questions:

  • since Rock beats Scissors, why would anyone ever choose Scissors?
  • since Paper beats Rock, why would anyone ever choose Rock?
  • since Scissors beats Paper, why would anyone ever choose Paper?

Reality isn’t binary. It’s not what’s “best”, it’s what’s fit for purpose in a given context and there are lots and lots of contexts out there.

This isn’t to say that all quick, simple interventions are wrong. If you find yourself in a house fire, more action and less comprehensive deliberation may well be in order. The key is matching the cost (largely in terms of time) of defining the problem space with cost (in terms of both effort and risk that the intervention adds to the problem) of crafting the solution.

Rock, Paper, Scissor, Lizard, Spock rules diagram

It’s almost guaranteed that the system contexts we deal with (both technical and social) will evolve toward more and more complexity. Surprises will emerge as a matter of course. We don’t need to make more by failing to take a more holistic view when we have the time to do so.

Management, Simple and Wrong – Semantics, Systems, and Self-Correction

Villain Caricature

Simple responses to complex situations are both seductive and dangerous. The difficulty in juggling lots of variables tempts us to employ abstraction so as to avoid being overwhelmed. Abraham Maslow’s observation, “I suppose it is tempting, if the only tool you have is a hammer, to treat everything as if it were a nail”, applies. Some things (e.g. landmines) react badly to being treated as if they were nails. Having more tools in the box may help avoid problems.

This isn’t the post I had in mind to write next, but it’s one that came about by accident (via a multi-day mass participant Tweet-storm, with my participation beginning here). I had planned an Organizations as Systems post re: multiple players in multiple contexts (competing, and possibly conflicting goals and motivations) and I stumbled into a conversation that should provide a nice preamble to that post which should follow this one.

Before I dive in, two quick notes:

  • Rather than try to summarize the entire conversation, I’m going to lay out what I brought to and took from it. There are far too many tweets and, as of this writing, I can’t be sure the conversation has concluded.
  • My thanks to everyone involved, whether named or not. This kind of civil, if contentious, dialog is much appreciated. When ideas rub together, it can produce irritation, but sometimes they also get polished.

Management is one of those things that, like landmines, tends to react badly to the hammer of simplistic thought. We can see this in managers who apply (or misapply) theories of management, particularly ones like scientific management (AKA Taylorism) to contexts where it is extremely inappropriate and counterproductive. Whether there really exists a context where Taylorism is actually appropriate or productive is a question for another day. We also see the hammer in reactions to abuses that dismiss all value of management out of hand. While the reaction is understandable, that doesn’t make it credible. The vicious circle just becomes more vicious; heat is generated but without corresponding light.

One thing that’s necessary to pin down is what we mean by the term “management”. Are we talking about a concept (“…the administration of an organization…”)? Are we talking about the job/profession? Perhaps the discipline (branch of knowledge) or academic discipline (field of study) is what we’re talking about. We could be talking about a theory management, or we could be talking about management practices, either individually or grouped into systems of management. Knowing specifically what’s being referred to is critical for evaluating statements. A very valid criticism of a specific theory or system (e.g. Taylorism) will likely fall apart when applied to the concept as a whole due to the fact that the concept is far broader and contrary examples are easily found.

Another issue relates to intent. Few would argue the universal detriment of poor management practices. Extracting the maximum possible effort from your employees is unlikely to result in the generation of the most value in the context of knowledge work. These practices are the very antithesis of fitness for purpose in that they do not materially benefit the organization and they alienate employees (which is yet another hit to the organization where the product is knowledge work). And yet, there are still managers that manage in that very manner. Are they, each and every one, evil? A simplistic answer, hard against either end of the spectrum, is almost surely going to be wrong. That being said, in my experience the distribution is skewed more towards the “no” side than not (just as I’ve found people who only perform when driven to it to be a very small minority).

Why would someone who wants to do their job well and in an ethical manner resort to practices that are harmful to all parties? With sadism eliminated as a motivation (there just aren’t enough in the population to account for all the positions to be filled), the far more plausible answer would be culture, tradition, and/or lack of knowledge regarding alternatives. In short, when the outcome of a system doesn’t match the intent, there’s a bug in the system.

The disconnect between leadership and management is also a problem. Leadership, admittedly, is a concept distinct from management. It makes sense that not every leader needs to be a manager. The extent to which we as a society tolerate management absent leadership, however, is shocking and part of the problem. Consider a tweet from Esther Derby:

I would argue that steering and enabling can be considered leadership qualities as much as management activities. There’s a place for supervision and compliance, however knowing how to achieve results without forcing the issue is, in my opinion, an extremely useful skill. This is not manipulation, rather a matter of understanding goals and how to achieve them intelligently. It’s a matter of understanding how to resolve potential conflicts between the goals and motivations of an organization, groups, and individuals and adapting the system so that the outcomes more closely track the intent. The alternative is allowing the system to degenerate into a web of perverse incentives that increase the gap between intent and outcome. This gap may benefit some individuals, but at the cost to other individuals and the organization as a whole.

Medicine is something that has been through a number of changes, large and small, by finding a way to adapt. While the concept of medicine (diagnosis, treatment, and prevention of disease and injury) has remained constant over time, the practices and theories have evolved greatly. The discipline itself has evolved so that not only does it adapt to change, but that it adapts in as optimal a manner as possible. In short, it has developed a culture of learning.

Understanding organizations from a systems standpoint means recognizing the need for sensing the fit between the system and its contexts (learning) and then steering to correct any mismatches (management). Simplistic approaches to management (particularly relatively static ones that have little save tradition to recommend them) can only lead to a widening gap between the intended outcome and actual results. At some point, this gap becomes wide enough to swallow the organization.

[Villain Carricature by J.J. via Wikimedia Commons.]

Square Pegs, Round Holes, and Silver Bullets

Werewolf

People like easy answers.

Why spend time analyzing and evaluating when you can just take some thing or some technique that someone else has already put to use and be done with it?

Why indeed?

I mean, “me too” is a valid strategy, right?

And we don’t want people to get off message, right?

And we can always find a low cost, minimal disruption way of dealing with issues, right?

I mean, after all, we’ve got data and algorithms, and stuff:

The thing is, actions need to make sense in context. Striking a match is probably a good idea in the dark, but it’s probably less so in daylight. In the presence of gasoline fumes, it’s a bad idea regardless of ambient light.

A recent post on Medium, “Design Sprints Are Snake Oil” is a good example. Erika Hall’s title was a bit click-baitish, but as she responded to one commenter:

The point is that the original snake oil was legitimate and effective. It ended up with a bad reputation from copycats who over-promised results under the same name while missing the essential ingredients.

Sprints are legitimate and effective. And now there is a lot of follow-up hype treating them as a panacea and a replacement for other types of work.

Good things (techniques, technologies, strategies, etc.) are “good”, not because they are innately right, but because they fit the context of the situation at hand. Those that don’t fit, cease being “good” for that very reason. Form absent function is just a facade. Whether it’s business strategy, management technique, innovation efforts, or process, there is no recipe. The hard work to match the action with the context has to be done.

Imitation might be the sincerest form of flattery, but it’s a really poor substitute for strategy.

Form Follows Function on SPaMCast 373

SPaMCAST logo

This week’s episode of Tom Cagley’s Software Process and Measurement (SPaMCast) podcast, number 373, features Tom’s essay on #NotImplementedNoValue and a Form Follows Function installment on simplistic mental models.

Tom and I discuss my post “All models may be wrong, but it’s not a contest to see how wrong you can be”, talking about cognitive biases and how overly simplistic mental models fail us.

You can find all my SPaMCast episodes using under the SPAMCast Appearances category on this blog. Enjoy!

All models may be wrong, but it’s not a contest to see how wrong you can be

HO Scale locomotive beside a pencil

The one thing you can be sure of is that nothing is dependent on only one thing.

Michael Feathers‘ tweet last week brought this to mind:

Too often we construct simplistic mental models that fail to account for outcomes that are possible, but inconvenient for us in some way. As Aneel noted while discussing OODA loops in his post “All Models Are Wrong Some Are Useful [In Some Context]”:

OODA is just a vehicle for the larger issue of models, biases, and model-based blindness — Taleb’s Procrustean Bed. Where we chop off the disconfirmatory evidence that suggests our models are wrong AND manipulate [or manufacture] confirmatory evidence.

Because if we allowed the wrongness to be true, or if we allowed ourselves to see that differentness works, we’d want/have to change. That hurts.

Furthermore:

Our attachment [and self-identification] to particular models and ideas about how things are in the face of evidence to the contrary — even about how we ourselves are — is the source of avoidable disasters like the derivatives driven financial crisis. Black Swans.

I wonder how many black swans are only “unpredictable” because we blind ourselves to possibilities.

It’s possible that we cannot eradicate self-deception. “Rats Can Be Smarter Than People” speaks to this via study results that found rats outperforming humans in one of a pair of learning tasks:

The first task involved rules. The second focused on information integration. Humans learn in both ways. Our rule-based system was an evolutionary development: How do you tell if a berry is good for eating? You learn that this small red one is good, and then you save energy by bypassing the ones of a different shape or color. So our brains have been conditioned to look for rules. We’re taught them in school, at work, and by our parents, and we can make many good decisions by applying the ones we’ve learned. But in other situations there’s too much going on for simple rules to work, and that’s when information integration learning has to kick in. Think of a radiologist evaluating an X-ray. If you ask him what rules he uses to determine whether a spot is cancer, he’d probably have a hard time verbalizing them. He’s learned from labeled examples in medical school and his own experience, and then developed an instinct for identifying cancerous spots based on what he’s seen before. Another example that comes to mind is a manager interviewing a job candidate. There aren’t any hard-and-fast rules about who will be a good hire. You have to consider many factors and rely on your judgment or on a gut feeling based on your experience with people in the workplace. Unfortunately, there’s a great deal of evidence showing that humans have a harder time learning how to integrate information in this way, because they seek rules even when there are none.

In other words, we have a model about learning (meta-model?) that works well in many situations. It works so well that we resist looking for context where it’s not the appropriate model. While this shows that a propensity for self-deception is natural, the fact that “self” is in there suggest that we have some control. Having some control obligates us to exercise what control we have and work to gain more.

Why?

A critical argument for the practice of software architecture is that the design of the solution must cohere with the problem space in order to be effective. In order to deliver decisions that achieve this goal, we need to be able to make sense of the problem space. Systems thinking, described by Tom Cagley as “…an approach to problem solving that emphasizes viewing problems as the output of the whole process, including the environment the system operates within”, is a technique to do so. The more we force ourselves to be aware of our biases and work to counteract them, the better our decisions will be.

Simple is good, but not when it’s too good to be true.

Who Needs Architects? Because Complexity Emerges

Why would you want to constrain creativity by controlling (Note: controlling does not necessarily imply dictating) the architecture of a system?

As Roger Sessions recently tweeted:

Another reason came out in an exchange between Roger and me:

Simplicity (“…as simple as possible, but not simpler”) is certainly a desirable system quality. Unnecessary complexity directly impairs maintainability and can indirectly affect qualities such as testability, security, extensibility, availability, and ease of deployment just to name a few. The coherent structure necessary to create and maintain simplicity when multiple people are involved is unlikely to happen accidentally. When it’s lacking, the result is often what Brian Foote and Joseph Yoder described in their 1999 classic “Big Ball of Mud”:

A BIG BALL OF MUD is haphazardly structured, sprawling, sloppy, duct-tape and bailing wire, spaghetti code jungle. We’ve all seen them. These systems show unmistakable signs of unregulated growth, and repeated, expedient repair. Information is shared promiscuously among distant elements of the system, often to the point where nearly all the important information becomes global or duplicated. The overall structure of the system may never have been well defined. If it was, it may have eroded beyond recognition.

Sixteen years later, this tendency to entropy via unnecessary complexity remains an issue. As Roger Sessions recently noted on his blog:

As IT systems increase in size, three things happen. Systems get more vulnerable to security breaches. Systems suffer more from reliability issues. And it becomes more expensive and time consuming to try to make modifications to those systems. This should not be a surprise. If you are like most IT professionals, you have seen this many times.

What you have probably not noticed is an underlying pattern. These three undesirable features invariably come in threes. Insecure systems are invariably unreliable and difficult to modify. Secure systems, on the other hand, are also reliable and easy to modify.

This tells us something important. Vulnerability, unreliability, and inflexibility are not independent issues; they are all symptoms of one common disease. It is the disease that is the problem, not the symptoms.

Kent Beck, in a post on FaceBook, “Taming Complexity with Reversibility”, recently noted the same:

As a system scales, whether it is a manufacturing plant or a service like ours, the enemy is complexity. If you don’t confront complexity in some way, it will eat you. However, complexity isn’t a blob monster, it has four distinct heads.

  • States. When there are many elements in the system and each can be in one of a large number of states, then figuring out what is going on and what you should do about it grows impossible.
  • Interdependencies. When each element in the system can affect each other element in unpredictable ways, it’s easy to induce harmonics and other non-linear responses, driving the system out of control.
  • Uncertainty. When outside stresses on the system are unpredictable, the system never settles down to an equilibrium.
  • Irreversibility. When the effects of decisions can’t be predicted and they can’t be easily undone, decisions grow prohibitively expensive.

If you have big ambitions but don’t address any of these factors, scale will wreck your system at some point.

Kent’s conclusion, “What changes–technical, organizational, or business–would you have to make to identify such decisions earlier and make reversing them routine?”, implies that addressing these factors requires systemic rather than localized response.

The counter-argument that’s frequently made is that the architecture can “emerge” from the implementation by doing the “simplest thing that can possibly work” and building on that. I touched briefly on this in my last post. To that, I’d add that not all changes are the same. Trying to bolt on fundamental, cross-cutting concerns (e.g. security, scalability, etc.) after the fact risks major (i.e. expensive) architectural refactoring. This type of refactoring is generally a hard sell and understandably so. That difficulty makes the Big Ball of Mud that much more likely.

This does not mean that the concept of emergence has no place in software architecture. As the various contexts making up the architecture of the problem are identified, challenges will emerge and need to be reconciled. With this naturally occurring emergence, it makes little sense to artificially generate challenges by refusing to “peek” at what’s ahead. As Ruth Malan has observed, architecture should be both “intentional and emergent”:

This need to contend with emergent issues continues for the entire lifetime of the system:

I’ve noted in the past that both planning and design share similarities. Regardless of the task, an appropriate design/plan provides a coherent direction that enhances the likelihood of success. Without it, Joe Dager’s question below is directly on point:

When Reality Gets in the Way – Applying Systems Thinking to Design

It’s easy to sympathize with this:

It’s also more than a little dangerous if our desire for simplicity moves us to act as if reality isn’t as complex as it is. Take, for example, a recent tweet from John Allspaw about over-simplification:

My observation in return:

As I noted in my previous post, it’s part of human nature to gravitate towards easy answers. We are conditioned to try to impose rules on reality, even when those rules are mistaken. Sometimes this is the result of treating symptoms in an ad hoc manner, as evidenced by this recent twitter exchange:

This goes by the name of the “balloon effect”, pressure on one area of the problem just pushes it into another in the way that squeezing a balloon displaces the air inside.

Sometimes our response is born of bias. In sociology, for example, this phenomenon has its own name: “normative sociology”:

The whole “normative sociology” concept has its origins in a joke that Robert Nozick made, in Anarchy, State and Utopia, where he claimed, in an offhand way, that “Normative sociology, the study of what the causes of problems ought to be, greatly fascinates us all”(247). Despite the casual manner in which he made the remark, the observation is an astute one. Often when we study social problems, there is an almost irresistible temptation to study what we would like the cause of those problems to be (for whatever reason), to the neglect of the actual causes. When this goes uncorrected, you can get the phenomenon of “politically correct” explanations for various social problems – where there’s no hard evidence that A actually causes B, but where people, for one reason or another, think that A ought to be the explanation for B.

Some historians likewise have a tendency to over-simplify, fixating on aspects that “ought to be” rather than determining what is (which is another way of saying what can be reasonably defended).

Decision-making is the essence of design. Thought processes that poorly match reality, whether due to bias or insufficient analysis or both, are unlikely to yield optimal results. Systems thinking, “…viewing ‘problems’ as parts of an overall system, rather than reacting to specific parts, outcomes or events, and thereby potentially contributing to further development of unintended consequences”, is an approach more likely to achieve a successful outcome.

When the end result will be a software system integrated into a social system (i.e. a system that is a component of an ecosystem), it makes sense to understand the problem space as the as-is system to be remediated. This holds true whether that as-is system is an automated one or not. While it is not feasible to minutely analyze the problem space, much less design in detail the solution, failing to appreciate the full context on a high level presents risks. These risks include not only those inherent in satisfying the needs of the overlooked context(s), but also those challenges that emerge from the interactions of the various contexts that make up the problem space.

Deciding on a particular design direction is, obviously, a decision. Deferring that determination is, likewise, a decision. Refusing to make a definite decision is a decision as well. The answer is not to push all decisions off to as late a date as possible, but to make decisions in the moment that are defensible given the information at hand. Looking at the problem space as a whole in the context of its ecosystem provides the perspective required to make the optimal decision.

“Microservice Mistakes – Complexity as a Service” on Iasa Global

I’m pleased to announce that I’ve been asked to continue as a contributor to the Iasa Global site. I’m planning to post original content there on at least a monthly basis. In the interim, please enjoy a re-post of “Microservice Mistakes – Complexity as a Service”