Form Follows Function on SPaMCast 426

SPaMCAST logo

One of the benefits of being a regular on Tom Cagley’s Software Process and Measurement (SPaMCast) podcast is getting to take part in the year-end round table (episode 426). Jeremy Berriault, Steve Tendon, Jon M. Quigley and I joined Tom for a discussion of:

  1. Whether software quality would be a focus of IT in 2017
  2. Whether Agile is over, at least as far as Agile as a principle-driven movement
  3. Whether security will be more important than quality and productivity in the year ahead

It was a great discussion and, as Tom noted, a great way to finish off the tenth year of the SPAMCast and kickoff year eleven.

You can find all my SPaMCast episodes using under the SPAMCast Appearances category on this blog. Enjoy!

Storming on Design

From Wikimedia: VORTEX2 field command vehicle with tornado in sight. Wyoming, LaGrange.

My youngest son has recently fallen in love with the idea of being a storm chaser when he gets older. Tweet storms are more my speed. There was an interesting one last week from Sarah Mei regarding the contextual nature of assessing design quality:

Context is a recurring theme for me. While the oldest post with that tag is just under three years old, a search on the term finds hits going all the way back to my second post in October, 2011. Sarah’s tweets resonated with me because in my opinion, ignoring context is a fool’s game.

Both encapsulation and silos are forms of separation of concerns. What differentiates the two is the context that makes the one a good idea and the other a bad idea. Without the context, you can come up with two mutually exclusive “universal” principles.

A key component of architectural design, is navigating the fractals that make up the contexts in which a system exists. Ruth Malan has had this to say regarding the importance of designing “outside the box”:

Russell Ackoff urged that to design a system, it must be seen in the context of the larger system of which it is part. Any system functions in a larger system (various larger systems, for that matter), and the boundaries of the system — its interaction surfaces and the capabilities it offers — are design negotiations. That is, they entail making decisions with trade-off spaces, with implications and consequences for the system and its containing system of systems. We must architect across the boundaries, not just up to the boundaries. If we don’t, we naively accept some conception of the system boundary and the consequent constraints both on the system and on its containing systems (of systems) will become clear later. But by then much of the cast will have set. Relationships and expectations, dependencies and interdependencies will create inertia. Costs of change will be higher, perhaps too high.

This interrelationship can be seen from the diagram taken from the same post:

System Context illustration, Ruth Malan

It’s important to bear in mind that contexts are multi-dimensional. All but the very simplest of systems will likely have multiple types of stakeholders, leading to multiple, potentially conflicting contexts. Accounting for these contexts while defining the problem and while designing a solution appropriate to the problem space is critical to avoiding the high costs Ruth referred to above.

Another takeaway is that context can (and likely will) change over time. Whether it’s changes in terms of staffing (as Sarah noted) or changes in the needs of users or changes in technology, a design that was fit for yesterday’s context can become unfit for today’s and a disaster for tomorrow’s.

Dealing with Technical Debt Like We Mean it

What’s the biggest problem with technical debt?

In my opinion, the biggest problem is that it works. Just like the electrical outlet pictured above, systems with technical debt get the job done, even when there’s a hidden surprise or two waiting to make life interesting for us at some later date. If it flat-out failed, getting it fixed would be far easier. Making the argument to spend time (money) changing something that “works” can be difficult.

Failing to make the argument, however, is not the answer:

Brenda Michelson‘s observation is half the battle. The argument for paying down technical debt needs to be made in business-relevant terms (cost, risk, customer impact, etc.). We need more focus on the “debt” part and remember “technical” is just a qualifier:

The other half of the battle is communicating, in the same business-relevant manner, the costs and/or risks involved when taking on technical debt is considered:

Tracking what technical debt exists and managing the payoff (or write off, removing failed experiments is a reduction technique) is important. Likewise, managing the assumption of technical debt is critical to avoid being swamped by it.

Of course, one could take the approach that the only acceptable level of technical debt is zero. This is equivalent to saying “if we can’t have a perfect product, we won’t have a product”. That might be a difficult position to sell to those writing the checks.

Even if you could get an agreement for that position, reality will conspire to frustrate you. Entropy emerges. Even if the code is perfected and then left unchanged, the system can rot as its platform ages and the needs of the business change. When a system is actively maintained over time without an eye to maintaining a coherent, intentional architecture, then the situation becomes worse. In his post “Enterprise Modernization – The Next Big Thing!”, David Sprott noted:

The problem with modernization is that it is widely perceived as slow, very expensive and high risk because the core business legacy systems are hugely complex as a result of decades of tactical change projects that inevitably compromise any original architecture. But modernization activity must not be limited to the old, core systems; I observe all enterprises old and new, traditional and internet based delivering what I call “instant legacy” [Note 1] generally as outcomes of Agile projects that prioritize speed of delivery over compliance with a well-defined reference architecture that enables ongoing agility and continuous modernization.

Kellan Elliot-McCrea, in “Towards an understanding of technical debt”, captured the problem:

All code is technical debt. All code is, to varying degrees, an incorrect bet on what the future will look like.

This means that assessing and managing technical debt should be an ongoing activity with a responsible owner rather than a one-off event that “somebody” will take care of. The alternative is a bit like using a credit card at every opportunity and ignoring the statements until the repo-man is at the door.

What We Have Here is a Failure to Communicate

“What we’ve got here is failure to communicate” and it appears to be epidemic. My own personal grand unified theory of everything is that most problems stem from or are aggravated by a lack of communication. Whether the topic is process, governance, planning, estimates, or design, chances are it’s easier to find opinions (and worse, policy and practices) based on one-sided viewpoints than a balanced understanding of the contexts involved. This is dangerous due to the simple fact that organizations are social systems (frequently fractal systems of systems) and as Ruth Malan has noted:

Russell Ackoff urged that to design a system, it must be seen in the context of the larger system of which it is part. Any system functions in a larger system (various larger systems, for that matter), and the boundaries of the system — its interaction surfaces and the capabilities it offers — are design negotiations. That is, they entail making decisions with trade-off spaces, with implications and consequences for the system and its containing system of systems. We must architect across the boundaries, not just up to the boundaries. If we don’t, we naively accept some conception of the system boundary and the consequent constraints both on the system and on its containing systems (of systems) will become clear later. But by then much of the cast will have set. Relationships and expectations, dependencies and interdependencies will create inertia. Costs of change will be higher, perhaps too high.

In other words, systems exist within an ecosystem, not a vacuum. Failure to take context into account harms systems (whether software or social) by baking in harmful structures and behaviors and we cannot take into account contexts that are not communicated and appreciated. This, by the way, is why you find posts about management and process on a site with the tagline “All Things Architectural”.

This is why I believe that successfully managing technical debt can’t happen without successfully communicating to the customer when it’s being taken on, what the costs involved are (or may be) and how it’s affecting the evolution of the product.

This is why I believe the answer to problems related to estimation lies in communication and collaboration, rather than #NoEstimates on the one hand or rigid authoritarianism on the other. This, in my opinion, holds true for all the social system issues (management, process, governance, planning, quality, and architectural design) that affect software development. Without understanding (which does not happen without communication) the goals behind the practices and what results are being achieved, it’s unlikely that the system will work to the satisfaction of anyone.

Local “optimizations” won’t fix systemic problems. We need to bridge the gaps. Can we talk?

Updated 4/8/2016 to fix a broken link.

“Dealing with “Technical Debt”” on Iasa Global

The term “technical debt” has a tendency to evoke an emotional response. Some people react puritanically – “technical debt” means sloppy code; sloppy code is sin; failing to call sin “sin” is condoning sin; to the stake with the heretic! Others will contend that technical debt solely refers solely to conscious trade-offs and by definition excludes code that is poorly written or designed.

Read “Dealing with “Technical Debt”” on Iasa Global for my take on how broadly “technical debt” should be defined.

Technical Debt – Why not just do the work better?

Soap or oil press ruins in Panayouda, Lesvos

A comment from Tom Cagley was the catalyst for my post “Design Communicates the Solution to a Problem”, and he’s done it again. I recently re-posted “Technical Debt – What it is and what to do about it” to the Iasa Global blog, and Tom was kind enough to tweet a link to it. In doing so, he added a little (friendly) barb: “Why not just do the work better?” My reply was “It’s certainly the best strategy, assuming you can“.

Obviously, we have a great deal of control over intentional coding shortcuts. It’s not absolute control; business considerations can intrude, despite our best advice to the contrary. There are a host of other factors that we have less control over: aging platforms, issues in dependencies, changing runtime circumstances (changes in load, usage patterns, etc.), and increased complexity due to organic growth are all factors that can change a “good” design into a “poor” one. As Tom has noted, some have extended the metaphor to refer to these as “Quality Debt, Design Debt, Configuration Management Debt, User Experience Debt, Architectural Debt”, etc., but in essence, they’re all technical deficits affecting user satisfaction.

Unlike coherent architectural design, these other forms of technical debt emerge quite easily. They can emerge singly or in concert. In many cases, they can emerge without your doing anything “wrong”.

Compromise is often a source of technical debt, both in terms of the traditional variety (code shortcuts) and in terms of those I listed above. While it’s easy to say “no compromises”, reality is far different. While I’m no fan of YAGNI, at least the knee-jerk kind, over-engineering is not the answer either. Not every application needs to be built for hundreds of thousands of concurrent users. This is compromise. Having insufficient documentation is a form of technical debt that can come back to haunt you if personnel changes result in loss of knowledge. How many have made a compromise in this respect? While it is a fundamental form of documentation, code is insufficient by itself.

A compromise that I’ve had to make twice in my career is the use of the shared database EAI pattern when an integration had to be in place and the target application did not have a better mechanism in place. While I am absolutely in agreement that this method is sub-standard, the business need was too great (i.e. revenue was at stake). The risk was identified and the product owner was able to make an informed decision with the understanding that the integration would be revised as soon as possible. Under the circumstances, both compromises were the right decision.

In my opinion, taking the widest possible view of technical debt is critical. Identification and management of all forms of technical debt is a key component of sustainable evolution of a product over its lifetime. Maintaining a focus on the product as a whole, rather than projects, provides a long-term focus should help make better compromises – those that are tailored to getting the optimal value out of the product. Having tunnel vision on just the code leaves too much room for surprises.

[Soap or oil press ruins in Panayouda, Lesvos Image by Fallacia83 via Wikimedia Commons.]

“Technical Debt – What it is and what to do about it” on Iasa Global Blog

Durer's Usury

In spite of all that’s been written on the subject of technical debt, it’s still a common occurrence to see it defined as simply “bad code”. Likewise, it’s still common to see the solution offered being “stop writing bad code”. Technical debt encompasses much more than that simplistic definition, so while “stop writing bad code” is good advice, it’s wholly insufficient to deal with the subject.

See the full post on the Iasa Global Blog (a re-post, originally published here).