Emergence versus Evolution

You lookin' at me?

Hayim Makabee’s recent post, “The Myth of Emergent Design and the Big Ball of Mud”, encountered a relatively critical reception on two of the LinkedIn groups we’re both members of. Much of that resistance seemed to stem from a belief that the choice was between Big Design Up Front (BDUF) and Emergent Design. Hayim’s position, with which I agree, is that there is continuum of design with BDUF and Emergent Design representing the extremes. His position, with which I also agree, is that both extremes are unlikely to produce good results, and that the answer lies in between.

The Wikipedia definition of Emergent Design cited by Hayim, taken nearly a word for word from the Agile Sherpa site, outlines a No Design Up Front (NDUF) philosophy:

With Emergent Design, a development organization starts delivering functionality and lets the design emerge. Development will take a piece of functionality A and implement it using best practices and proper test coverage and then move on to delivering functionality B. Once B is built, or while it is being built, the organization will look at what A and B have in common and refactor out the commonality, allowing the design to emerge. This process continues as the organization continually delivers functionality. At the end of an agile or scrum release cycle, Development is left with the smallest set of the design needed, as opposed to the design that could have been anticipated in advance. The end result is a smaller code base, which naturally has less room for defects and a lower cost of maintenance.

Rather than being an unrealistically extreme statement, this definition meshes with ideas that people hold and even advocate:

“You need an overarching vision, a “big picture” design or architecture. TDD won’t give you that.” Wrong. TDD will give you precisely that: when you’re working on a large project, TDD allows you to build the code in small steps, where each step is the simplest thing that can possibly work. The architecture follows immediately from that: the architecture is just the accumulation of these small steps. The architecture is a product of TDD, not a pre-designed constraint.

Portion of a comment to Dan North’s “PUBLISHED: THE ART OF MISDIRECTION”

Aspects of a design will undoubtedly emerge as it evolves. Differing interpretations of requirements as well as information deficits between the various parties, not to mention changing circumstances all conspire to make it so. However, that does not mean the act of design is wholly emergent. Design connotes activity whereas emergence implies passivity. A passive approach to design is, in my opinion, unlikely to succeed in resolving the conflicts inherent in software development. In my opinion, it is the resolution of those conflicts which allows a system to adapt and evolve.

I’ve previously posted on the concept of expecting a coherent architecture to emerge from this type of blinkered approach. Both BDUF and NDUF hold out tremendous risk of wasted effort. It is as naive to expect good results from ignoring information (NDUF) as it is to think you possess all the information (BDUF). Assuming a relatively simple system, ignoring obvious commonality and obvious need for flexibility in order to do the “simplest thing that could possibly work, then refactor” guarantees needless rework. As the scale grows, the likelihood of conflicting requirements will grow. Resolving those conflicts after code for one or more features is in place will be more likely to yield unsatisfactory compromises.

The biggest weakness of relying on refactoring is that there are well-documented limits to what people can process. As the level of abstraction goes down, the number of concerns goes up. This same limit that dooms BDUF to failure limits the ability to refactor large systems into a coherent whole.

Quality of service issues are yet another problem area for the “simplest thing that could possibly work” method. By definition, that concentrates on functionality to the exclusion of non-functional concerns. Security and scalability are just two concerns that typically fare poorly when bolted on after the fact. Premature optimization is to be avoided, but being aware of the expected performance environment can help you avoid blind alleys.

One area where I do agree with the TDD advocate quoted above, is that active design imposes constraints. The act of design involves defining structure. As Ruth Malan has said, “negative space is telling; as is what it places emphasis on”. Too little structure poses as much risk as too much.

An evolutionary design process, such as Hayim’s Adaptable Design Up Front (ADUF), recognizes the futility of predicting the future in minute detail (BDUF) without surrendering to formlessness (NDUF). Experience about what parts of a system are most likely to change is invaluable. Coupled with reasonable planning based on what is known about the big picture of the current release and what’s known about follow-up releases can be used to drive a design that strikes the right balance – flexible, without being over-engineered.

[Photograph by Jose Luis Martinez Alvarez via Wikimedia Commons.]


16 thoughts on “Emergence versus Evolution

  1. Terrific post Gene! In the context a a large system starting from scratch, I assert that between the two extremes, NDUF is worse on the budget & schedule than BDUF. At least with BDUF, you have a chance of getting at least the large grained abstractions (those concerned with NFRs) “somewhat” in the ballpark. With NDUF, the chance of satisfying NFRs by coding upwards from concrete to abstract is virtually nil.


    • Thanks, Tony. Agreed – coherent, with issues is going to be easier to fix than incoherent and inconsistent (as the number of uncoordinated decisions increase the entropy is going to explode). I didn’t get into dependency management with this post, but I can see that as big factor as well.


  2. Great article. We don’t call it ADUF (although we should), but you’re describing almost exactly how design is handled in our dev teams. The initial design is an overview and, if necessary, class stubs to show how they are initially expected to interact. Then, as development continues, we discuss the design, how it’s working out, and how proposed changes might fit into the big picture.

    We don’t pretend to have The Perfect Design(TM) from the start, but we do give direction.

    As an aside, I never understood BDUF. How do you eat an elephant? One bite at a time.

    And with Barbecue sauce.


    • I’d ask whether the sauce is vinegar-based or tomato-based, but that really risks a religious war ;-).

      Agreed – whether it’s ADUF or Just Enough Design Up Front, the core comes down to matching the level of detail with the level of understanding. IMO, BDUF runs off the rails because people are trying to make decisions without enough (any) information. By the same token, ignoring what you do know because it doesn’t mesh with your technique seems like a loser as well.


  3. Pingback: Bumper Sticker Philosophy | Form Follows Function

  4. Pingback: Design by Committee | Form Follows Function

  5. I guess this is why the road system in SC looks like a game of pick-up-sticks. Even if they tried to “design it up front”, hundreds of years ago what would they have to go on? In MI, it looks like a grid, but the rivers, towns, railroads and cities all made a mess of it. Too bad.
    If we can’t even figure out roads after having them for thousands of years, how can we do software?


  6. Pingback: Uber-Architects: The Building Metaphor Is Dead | DaedTech

  7. Pingback: Why does software development have to be so hard? | Form Follows Function

  8. Pingback: Software Development, Coding, Forests and Trees | Form Follows Function

  9. Pingback: Emergence versus Evolution | Iasa Global

  10. Pingback: Accidental Architecture | Form Follows Function

  11. Pingback: Bumper Sticker Philosophy | Iasa Global

  12. Pingback: Design By Committee | Iasa Global

  13. Pingback: Bumper Sticker Philosophy | IasaGlobal

  14. Pingback: Designs Don't Emerge - DaedTech

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.