Emergent Architectures and Growing Pains

Occasionally I hear people still raising the agile mantra against Big Design/Requirements Up Front. The thing is that Agile Manifesto never said to intentionally bury your head in the sand with regards to the purpose of the system. It was a push-back against spending months in analysis without anything but documents coming out, but the goal was to reach a middle ground. Nobody ever said “no design up-front” or “no requirements up front”.
(Udi Dahan from “A CQRS Journey – with and without Microsoft”)

In my premier post, I touched on the concept of emergent architecture in passing, noting that Wikipedia’s definition of architecture should be amended to read “Architecture should be both the process and product of planning, designing and construction, although it may just be the product of construction.” (my additions in bold). In essence, an architecture will develop regardless of whether that development is managed or not. The key question is whether the architecture provides value or headaches.

Lately the topic has been a popular one. A discussion on The Enterprise Architecture Network group on LinkedIn was active all last week. This weekend Igor Lobanov posted an excellent take on the subject, titled “Can an architecture emerge?”. Raja Bavani just posted “Can Software Projects Mature by Accident?”. It even made its way into the comments of the post where Dan North announced that he had reprinted “The Art of Misdirection” on his site. The person commenting took issue with a limitation of Test Driven Development (TDD) identified by Dan North:

“You need an overarching vision, a “big picture” design or architecture. TDD won’t give you that.” Wrong. TDD will give you precisely that: when you’re working on a large project, TDD allows you to build the code in small steps, where each step is the simplest thing that can possibly work. The architecture follows immediately from that: the architecture is just the accumulation of these small steps. The architecture is a product of TDD, not a pre-designed constraint.

It should come as little surprise that I find the concept of emergent architecture highly suspect, and the attitude in the quote above illustrates why. The “simplest thing that can possibly work” may or may not be sufficiently flexible. If that need for flexibility can be foreseen, then the “simplest thing that can possibly work” represents wasted time and effort. Likewise, an architecture that is “just the accumulation of these small steps” is unlikely to be robust or cohesive. It is extremely naive to expect a system’s requirements to exist without conflict. Working piecemeal, I find it unlikely that these natural tensions (such as security versus usability and performance versus simplicity) will be adequately addressed. Cross-cutting concerns ensure that a system must be more than the sum of its individual parts.

Igor Lobanov’s post compares the organic evolution of a system with Darwinian evolution, which is a compelling analogy. Left to its own devices, a solution will accumulate technical debt. Techniques such as TDD may allow the system to remain fit for purpose on a feature level, but the overall flexibility and robustness of the system will suffer. The more complex the system, the less amenable it will be to emergent architecture.