Architecting the Process

Jimmy Bogard has moved on to Lean from Scrum. InfoQ asks “Is Kanban the new Scrum?”, while Jim Bird asks “What can you get out of Kanban?”. Dan North touched a raw nerve by suggesting that techniques like TDD should be used only when they add value rather than universally. A post titled “Seven Things I Hate About Agile” is answered by another named “7 things I hate about people that don’t know Agile”.

bettyfjord, commenting on Jimmy Bogard’s “Why I’m done with Scrum”, probably spoke for many with the observation:

As a dev with many years’ experience delivering and responding in very high-pressure environments, I still cannot believe how seriously this methodology stuff gets debated. Y’all clearly don’t have enough work to do.

Certainly the process wars have consumed countless hours and electrons that could otherwise have been spent producing software. That being said, if we don’t make time to talk about how we work, when does it get evaluated? If our processes aren’t evaluated, how do we know they’re optimal, or at least moving in that direction? If we’re taking it on faith that our process is the one true way to develop, how agile is that?

At first glance, certainty sounds like a good thing. According to Dictionary.com, to be certain is to be “free from doubt”. Doubt, however, can be a powerful driver of innovation. Over the centuries, people have from time to time held the opinion that progress has reached its end point. Their certainty has, without exception, been wrong and the doubters right.

Even when we find that our processes serve our needs, the evaluation still provides value in the form of validation. Unexamined processes risk becoming psychological experiments proving that groups will adhere to practices long after any need exists, just because “that’s the way we do it”.

Rather than expecting one methodology to suffice for all, it seems much more reasonable to accept that organizations, their systems and the focus of their development teams will differ. An independent software vendor (ISV) will have different circumstances and priorities to deal with than an in-house corporate development team who will operate under constraints different from contractors. Within these categories, goals and priorities can differ greatly. A startup with the next great game will almost certainly operate differently from a company selling financial software. For either to use the same process as the other would be a disaster.

The working environment can likewise vary within an organization. Teams working on innovative web initiatives for an enterprise would most likely be severely hampered by the restrictions the financial system teams work under. Likewise, allowing the financial systems teams the same latitude given the R&D groups could carry both professional and legal consequences. Even teams working on different aspects of the same product could benefit from different processes. Time-boxed methods may work better for those doing new development, particularly on products that can’t be continuously deployed. At the same time, those providing support for the same product may benefit from processes that maximize delivery.

Bogard’s situation illustrates another reason to favor a pragmatic approach. Technologies, people and organizations all change. Just because a particular process fits today’s situation does not mean that it will continue to work in the future. Constant evaluation and refinement is key to remaining relevant. As Jim Highsmith has noted:

There is no Agility for Dummies. Agility isn’t a silver bullet. You don’t achieve it in five easy steps. So what is it? For myself, I’ve characterized agility in two statements:

Agility is the ability to both create and respond to change in order to profit in a turbulent business environment.

Agility is the ability to balance flexibility and stability.

Part of responding to change and remaining flexible is avoiding letting processes calcify. From another Highsmith post:

The Agile Manifesto was carefully worded to promote adaptability—to steer people away from rigid interpretations of the principles. Take the delivery principle for example, “Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.” It would have been much easier to write this principle as “Deliver working software every two weeks,” but that wording would lead people to rigidity (they get there on their own anyway).

Furthermore, what happens in too many organizations is that practices become static and then quietly elevated to the level of principle—something that can’t be violated. A good practice, “daily standups,” becomes bureaucratic when it is interpreted as “You must do daily standups, or else!” When questioned, the proponents of this newly crowned principle usually respond with “well, Joe Jones who wrote the book on Agile says so on page 48.” They lose track of why they are using the practice and it becomes a de facto standard rather than a guideline.

In the past, I’ve noted that “why” is the most important question architects should ask themselves when designing. Likewise, when designing a process, we should ask what we’re trying to accomplish and whether the processes we’re adopting will further those goals.

Rules are rules…or are they?

Elizabeth: Wait! You have to take me to shore. According to the Code of the Order of the Brethren…

Barbossa: First, your return to shore was not part of our negotiations nor our agreement so I must do nothing. And secondly, you must be a pirate for the pirate’s code to apply and you’re not. And thirdly, the code is more what you’d call “guidelines” than actual rules. Welcome aboard the Black Pearl, Miss Turner.

(from “Pirates of the Caribbean: The Curse of the Black Pearl”)

Over the course of a career in technology, you collect “rules” that guide the way you work. However, as a blogger noted: “Good design principles are usually helpful. But, are they always applicable?”.

We know redundant data is bad, except for caching, when it’s good. Database normalization is a good thing until it affects performance, then it’s not so good…same with transactions. Redundant code, now there’s an absolute! Except we find that the Open/Closed Principle dictates that new code be added for changes, rather than modifying the existing code. What to do?

The Open/Closed Principle always bothered me. I agree with it philosophically–good designs make it possible to add functionality without disturbing existing features–but in my experience there are no permanently closed abstractions. Superclasses or APIs might be stable for a (relatively) long time, but eventually even the most fundamental classes and interfaces need updating to meet emerging needs.

(Kent Beck, “The Open/Closed/Open Principle”)

Even Bob Martin, who stated “In many ways this principle is at the heart of object oriented design” recognized its limits:

It should be clear that no significant program can be 100% closed… In general, no matter how “closed” a module is, there will always be some kind of change against which it is not closed.

(from “The Open-Closed Principle”)

The takeaway is that an architect should be pragmatic, not dogmatic. Understanding the “why” behind a “rule” allows you to know when it doesn’t apply. Knowing the trade-offs allows you to make informed, rationale decisions that are consistent with the needs at hand. Blindly adhering to received wisdom is just magical thinking, and it’s value is more a function of chance than reality. By the same token, understanding why you’re acting contrary to common practice marks the difference between boldness and gambling.

A foolish consistency is the hobgoblin of little minds, adored by little statesman and philosophers and divines. With consistency a great soul has simply nothing to do. He may as well concern himself with his shadow on the wall. Speak what you think now in hard words, and to-morrow speak what to-morrow thinks in hard words again, though it contradict every thing you said to-day.

(Ralph Waldo Emerson, “Self Reliance”)