This week’s episode of Tom Cagley’s Software Process and Measurement (SPaMCast) podcast, number 365, features Tom’s essay on Agile Project Charters, Kim Pries on improvisation in software development, and a Form Follows Function installment on customer-driven development.
Customer-driven development refers to my post “Maybe It’s Time for Customer Driven Development”, where I discussed the need to consider the impact on the customer when making changes to development processes.
Analogies are never perfect, but reading Erik Dietrich’s “Do Programmers Practice Computer Science?” brought one to mind. Software development has much in common with the practice of medicine. Software development, like medicine, involves the application of knowledge. Also like medicine, this application is made complex by considerations of context. Yet another commonality is that in both disciplines, there are (or, at least, should be, limits regarding experimentation).
Erik’s post used the following comparison of developers to electricians:
Let’s consider three actors in the realm of physics, as a science.
- A physicist, who runs electricity through things to see if they explode.
- An electrical engineer, who takes the knowledge of what explodes from the physicist and designs circuitry for houses.
- An electrician, who builds houses using the circuits designed by the electrical engineer.
I list these out to illustrate that there are layers of abstraction on top of actual science. Is an electrician a scientist, and does the electrician use science? Well, no, not really. His work isn’t advancing the cause of physics, even if he is indirectly using its principles.
Let’s do a quick exercise that might be a bit sobering when we think of “computer science.” We’ll consider another three actors.
- Discrete mathematician, looking to win herself a Fields medal for a polynomial time factoring algorithm.
- R&D programmer, taking the best factoring algorithms and turning them into RSA libraries.
- Line of business programmer, securing company’s Sharepoint against script kiddies uploading porn.
Programming is knowledge work and non-repetitive, so the comparison is unfair in some ways. But, nevertheless, what we do is a lot more like what an electrician does than what a scientist does. We’re not getting paid to run experiments — we’re getting paid to build things.
There is definitely some validity in this. The three roles in each example have many similarities. His observation that development work is “non-repetitive”, however, is key. Electricians work in a more certain context than doctors who may need to account for body chemistry or metabolism. Likewise, developers may find environmental factors (e.g. memory usage profile, network load, etc.) produce uncertainty in the course of their work. Whereas the plumbing and electrical systems in a house are mostly separate, biological systems and information systems tend to be more intertwined.
Another similarity between software development and the practice of medicine is the feedback loop. The physicist will never hear back from the electrician, but physicians doing research are not similarly removed from practitioners. Practice and theory in medicine have a chicken and egg relationship where neither is clearly dominant, but each influences the other. Likewise with software development. Ethics and practicality in both cases constrain pure research.
As Erik noted, developers are “…not getting paid to run experiments — we’re getting paid to build things”. That being said, the uncertainties mean that, like physicians, we can’t be positive about the exact outcome without trying a particular course of action (which isn’t really an experiment):
Like doctors, those involved in software development have an ethical obligation to let our “patients” know when we’re learning on the job and what the risks are (not to mention the obligation to try things that are in their best interests and not just something we want to test drive). In addition to considerations of professionalism, more open communication has its benefits. We can solve problems and advance the practice at the same time.
Please take a moment to check out the page I just posted, “Do You Have Two Minutes?”. It tells about an incredible young man who is battling cancer.
I hope you’ll consider helping him out, but regardless, the time spent hearing about him would be well spent.
-Gene
In my last post, “Locking Down the Prisoners: Control, Conflict and Compliance for Organizations”, I returned to a topic that I’ve been touching on periodically over the last year, organizations as systems, which overlaps significantly with the topic of enterprise architecture (not to be confused with enterprise IT architecture of which EA is a superset). While I do not pretend to be an expert on the subject, the fractal nature of the environment I work within as a software architect makes it difficult (perhaps even dangerous) to ignore.
Systems reside within ecosystems, which being systems themselves, reside within an ecosystem of their own. Both systems and ecosystems are mutually influencing and that influence must be understood to the extent possible and accounted for, both upstream and down. Where this fails to happen, coherence between information systems and social systems breaks down. Significantly, issues in systems at higher levels of granularity can negate positive aspects of the systems that comprise them.
This is probably a good place to point out that my use of terms like “social system” is not meant to remove the human aspect. On the contrary, social systems are intensely human in nature. Where those systems are dysfunctional, it is individuals that ultimately pay the price. The intent, is to point out the inter-relationships that make up our environment.
Consider the hypothetical systems outlined in “Making and Taming Monoliths”. Assuming all of the systems involved were modular enough and technically capable of interoperation, it would not be enough. Considerations of data architecture (starting with, which system is authoritative?) could spike the effort, or at least seriously delay it. Organizational structure (aka Conway’s Law), behavior (management, governance, process), and psychology/culture could all either impel or impede. A perfect example of this concept is the near requirement that DevOps be in place where microservices are used – it may be possible to develop and deploy microservices in other process models, but it will involve far more difficulty.
“Alignment” is a term that is often mentioned in terms of IT. However, even if an organization’s IT systems are perfectly aligned with the business units they support, it will do little good if those units are working at cross purposes. Just as the full stack of an application requires coherent, intentional design to work well, so too does an enterprise. This does not, however, imply that a rigid, top-down mode of operation is needed; it’s actually quite the opposite.
In “Auftragstaktik and fingerspitzengefühl”, Tom Graves described the UK’s World War II air defense system as an example of an effective enterprise. The two German words refer to the concepts of situational awareness (fingerspitzengefühl) and mission tactics (Auftragstaktik), which underlay the success of the system. In essence, situational awareness at each level was enabled by the flow of information up and down the hierarchy, while decisions appropriate to each level were made at that level informed by understanding of the situation and the commander’s intent. This is not just a matter of process. As Tom noted:
Another key element is organisational-culture – whether the culture invites or dissuades individual judgement within real-time action (auftragstaktik), elicitation and capture of real-world subtleties (for fingerspitzengefühl) and/or whistleblower-type algedonic responses.
An organization must not only be structured so as to achieve its aims, the drive to do so must be part of its DNA. Even when some components of the organization may have remits that conflict with others (think audit, accounting, InfoSec, etc.), the ultimate direction of all parts should harmonize. Structuring a system so as to resolve conflicts is an architectural practice, regardless of the type of system.
The most important thing to learn about management and governance is knowing when and how to manage or govern and more importantly, when not to.
The story is told about a very new and modern penal facility, the very epitome of security and control. Each night, precisely at 11:00 PM, the televisions were shut off and the inmates were herded into their cells for lights out. Since the inmates tended to dislike their enforced bedtime, fights would ensue during the lockdown and throughout the night when the cells needed to be opened (both for purposes of head counts and to respond to the inevitable conflicts caused by locking people in close quarters). If the problems were pervasive enough, an entire housing unit might be punished by – wait for it – being confined to their cells (perpetuating the cycle).
Management of the facility was at a loss on what to do. The conflict was causing disruption in the day-to-day activities. This disruption further exacerbated tensions. The fights led to injuries to both staff and inmates, raising costs and risk of civil litigation, as well as causing staffing problems.
The answer was simple – stop the lockdowns. When the policy was reviewed objectively, it was obvious that enforcement was yielding no benefits to offset the many costs. In fact, stopping enforcement actually increased security by reducing tensions and causing the night owls to sleep in during the day. In a real-life zen moment, it was realized that letting go of the illusion of control provided real control (or at least something closer to it).
Most organizations could benefit from a similar epiphany.
This is not to suggest that process, management, and governance are unnecessary, far from it. Instead, it’s important that the system by which things are run is…systemic. As Tom Graves likes to say, “…things work better when they work together, on purpose”. Intentional design applies to social systems, just as it applies to software systems. Ad hoc evolution, by way of disjointed decisions unencumbered with any coherence, lead to accidental structures. Entropy emerges.
This can be seen in a tweet from Charles T. Betz:
Or, as Gary Hamel tweeted:
The alternative is to do as Yves Morieux stated in his TED talk: “We need to create organizations in which it becomes individually useful for people to cooperate.” This involves a ruthless attention to cause and effect. This involves creating environments where unnecessary friction is removed and necessary friction is understood to be necessary by all involved. It’s a lot easier to get compliance when it’s easier to comply and a lot easier to get conflict when you provoke it.
The signature benefit of a microservice architecture is that its highly granular nature allows for a great deal of flexibility in composing applications. Components are simplified by virtue of a high degree of focus. The ability to replace individual components is enhanced by the modularity inherent in the style.
A very significant drawback to microservice architecture is that its highly granular nature can lead to a great deal of complexity in composing applications. Highly focused components can force service consumers to become more involved in the internals of an interaction than they might otherwise wish. Unwanted options can become more of a source of confusion than useful modularity.
How do you resolve this paradox? See the full post on the Iasa Global Site
“What we’ve got here is failure to communicate” and it appears to be epidemic. My own personal grand unified theory of everything is that most problems stem from or are aggravated by a lack of communication. Whether the topic is process, governance, planning, estimates, or design, chances are it’s easier to find opinions (and worse, policy and practices) based on one-sided viewpoints than a balanced understanding of the contexts involved. This is dangerous due to the simple fact that organizations are social systems (frequently fractal systems of systems) and as Ruth Malan has noted:
Russell Ackoff urged that to design a system, it must be seen in the context of the larger system of which it is part. Any system functions in a larger system (various larger systems, for that matter), and the boundaries of the system — its interaction surfaces and the capabilities it offers — are design negotiations. That is, they entail making decisions with trade-off spaces, with implications and consequences for the system and its containing system of systems. We must architect across the boundaries, not just up to the boundaries. If we don’t, we naively accept some conception of the system boundary and the consequent constraints both on the system and on its containing systems (of systems) will become clear later. But by then much of the cast will have set. Relationships and expectations, dependencies and interdependencies will create inertia. Costs of change will be higher, perhaps too high.
In other words, systems exist within an ecosystem, not a vacuum. Failure to take context into account harms systems (whether software or social) by baking in harmful structures and behaviors and we cannot take into account contexts that are not communicated and appreciated. This, by the way, is why you find posts about management and process on a site with the tagline “All Things Architectural”.
This is why I believe that successfully managing technical debt can’t happen without successfully communicating to the customer when it’s being taken on, what the costs involved are (or may be) and how it’s affecting the evolution of the product.
This is why I believe the answer to problems related to estimation lies in communication and collaboration, rather than #NoEstimates on the one hand or rigid authoritarianism on the other. This, in my opinion, holds true for all the social system issues (management, process, governance, planning, quality, and architectural design) that affect software development. Without understanding (which does not happen without communication) the goals behind the practices and what results are being achieved, it’s unlikely that the system will work to the satisfaction of anyone.
Local “optimizations” won’t fix systemic problems. We need to bridge the gaps. Can we talk?
Updated 4/8/2016 to fix a broken link.
If you think building a system is challenging, try maintaining one.
Tom Cagley‘s recent post “Plan to Throw One Away Re-Read Saturday: The Mythical Man-Month, Part 11”, was a good reminder that while “technical debt” may be something currently on the radar for many, it’s far from a new phenomenon. The concept of instant legacy applications was in place when forty years ago when Frederick Brooks wrote his masterpiece, even if they weren’t called that. As Tom observed in the post:
Rarely is the first attempt useful to the end consumer, and the usefulness of that first attempt is less in the code than in the feedback it generates. Software development is no different. The initial conceptual design and anticipated technical architecture of a large project rarely stands up to the rigors of the discovery process, and those designs should be learned from and then thrown away.
The faulty assumptions and design flaws accumulate not only from sprint to sprint leading up to the initial release, but also from release to release. In spite of the fact that a product can be so seriously flawed, throwing it away and starting over is easier said than done. While sunk costs cannot be recovered, too sanguine an attitude towards them may not enhance your credibility with the customer. Having to pay for the same thing over and over can make them grumpy.
This sets up a dilemma, one that frequently leads to living with technical debt and attempting to incrementally patch it up. There are limits, however, to the number of band-aids that can be applied. This might make it tempting to propose a rewrite, but as Erik Dietrich stated in “The Myth of the Software Rewrite”:
Sure, they know things now that they didn’t know when they started on this code 3 years ago. But won’t the same thing be true in 3 years? Won’t the developers then be looking at the code and saying, “this is a mess — if only we knew in 2015 what we now know in 2018!” And, beyond that, what makes you think that giving the same group of people the same marching orders won’t result in the same kind of code?
The “big rewrite from scratch because this is a mess” is a losing strategy.
Fortunately, there is an alternative. Quoting Tom Cagley again from the same post as above:
If change is both inevitable and good (within limits), then both systems and organizations (a type of system) need to be engineered to support and facilitate change. Architecturally, techniques such as modularization, object-oriented design and other processes that foster simplification and incremental change create an environment in which change isn’t avoided, but rather encouraged.
While we may laugh at the image of changing a tire while the vehicle is in motion, it is an accurate metaphor. Customers expect flexibility and change on the go; waiting equals lost business. The keys to evolving in place are having an intentionally designed, modular architecture and an understanding of where the weaknesses lie. Both of these are concerns that reside squarely on the architect’s plate.
Modularity not only makes an application more easily maintainable via separation of concerns, but it also embraces change by making components replaceable. This is one of the qualities that has made microservices such a hot topic, although it would be a mistake to think that microservices are the only way (or best way in all cases) to achieve modularity.
Modularity brings benefits beyond the purely technical as well. Rewrites of a fraction of an application are more easily sold than big-bang efforts. Demonstrating forethought (while you can’t predict what the change will be, predicting the need for change is more of a sure thing) demonstrates concern for the customer’s welfare, which should make for a better relationship.
Being able to throw a system away a little at a time allows us to keep the car on the road while it changes and adapts to changing conditions.
Form Follows Function 