Software Development, Coding, Forests and Trees

They say there's a forest in here somewhere

Some responses to my post “Why does software development have to be so hard?” illustrated one major (in my opinion) aspect of the problem – for many people, software development is synonymous with coding. It’s certainly understandable that someone might jump to that conclusion. After all, no matter how many slides, documents, diagrams, etc. someone produces, it is code that makes those ideas real.

Code, however, is not enough.

Over the last seventeen-plus years that I’ve been involved in software development, great strides have been made in languages and platforms. Merely look at the plumbing code needed to write a Hello World for Windows in C should you need convincing. Frameworks for application infrastructure, unit testing and acceptance testing are plentiful. Coding and coding cleanly is far, far easier and yet, people still complain about software.

While poor quality code can sink a product, excellent quality code cannot make a product. No matter how right you build a thing, the customer won’t be happy if it’s the wrong thing. The Hagia Sophia, Taj Mahal, Empire State Building, and many others are all breathtakingly magnificent structures that would utterly fail a customer who wanted (not to mention, budgeted for) a garage. We still fail to adequately understand the needs of our customers and the environments they work within. This is an area that desperately needs improvement. This is not a technical issue, but one of communication, collaboration, and organization. Neither customer nor provider can impose this improvement unilaterally.

Understanding the architecture of the problem is critical to designing and evolving the architecture of the solution, which is yet another area of need. Big Design Up Front (BDUF) assumes too much certainty and never (at least in my experience) survives contact with reality. No Design Up Front (NDUF), however, swings too far in the opposite direction and is unlikely to yield a cohesive design without far too much re-work. Striking a balance between the two is, in my opinion, key to producing an architecture that satisfies the functional and quality of service requirements of today while retaining sufficient flexibility for tomorrow.

Quality code implementing an architectural design grounded in a solid understanding of the customer’s problem space is, in my opinion, the essence of software development. Anything less than those three elements misses the mark.

Why does software development have to be so hard?

Untangling this could be tricky

A series of 8 tweets by Dan Creswell paints a familiar, if depressing, picture of the state of software development:

(1) Developers growing up with modern machinery have no sense of constrained resource.

(2) Thus these developers have not developed the mental tools for coping with problems that require a level of computational efficiency.

(3) In fact they have no sensitivity to the need for efficiency in various situations. E.g. network services, mobile, variable rates of change.

(4) Which in turn means they are prone to delivering systems inadequate for those situations.

(5) In a world that is increasingly networked & demanding of efficiency at scale, we would expect to see substantial polarisation.

(6) The small number of successful products and services built by a few and many poor attempts by the masses.

(7) Expect commodity dev teams to repeatedly fail to meet these challenges and many wasted dollars.

(8) Expect smart startups to limit themselves to hiring a few good techies that will out-deliver the big orgs and define the future.

The Fallacies of Distributed Computing are more than twenty years old, but Arnon Rotem-Gal-Oz’s observations (five years after he first made them) still apply:

With almost 15 years since the fallacies were drafted and more than 40 years since we started building distributed systems – the characteristics and underlying problems of distributed systems remain pretty much the same. What is more alarming is that architects, designers and developers are still tempted to wave some of these problems off thinking technology solves everything.

Why?

Is it really this hard to get it right?

More importantly, how do we change this?

In order to determine a solution, we first have to understand the nature of the problem. Dan’s tweets point to the machines developers are used to, although in fairness, those of us who lived through the bad old days of personal computing can attest that developers were getting it wrong back then. In “Most software developers are not architects”, Simon Brown points out that too many teams are ignorant of or downright hostile to the need for architectural design. Uncle Bob Martin in “Where is the Foreman?”, suggests the lack of a gatekeeper to enforce standards and quality is why “our floors squeak”. Are we over-emphasizing education and underestimating training? Has the increasing complexity and the amount of abstraction used to manage it left us with too generalized a knowledge base relative to our needs?

Like any wicked problem, I suspect that the answer to “why?” lies not in any one aspect but in the combination. Likewise, no one aspect is likely, in my opinion, to hold the answer in any given case, much less all cases.

People can be spoiled by the latest and greatest equipment as well as the optimal performance that comes for working and testing on the local network. However, reproducing real-world conditions is a bit more complicated than giving someone an older machine. You can simulate load and traffic on your site, but understand and accounting for competing traffic on the local network and the internet is a bit more difficult. We cannot say “application x will handle y number of users”, only that it will handle that number of users under the exact same conditions and environment as we have simulated – a subtle, but critical difference.

Obviously, I’m partial to Simon Brown’s viewpoint. The idea of a coherent, performant design just “emerging” from doing the simplest thing that could possibly work is ludicrous. The analogy would be walking into an auto parts store, buying components individually, and expecting them to “just work” – you have to have some sort of idea of the end product in mind. On the other hand, attempting to specify too much up front is as bad as too little – the knowledge needed is not there and even if it were, a single designer doesn’t scale when dealing with any system that has a team of any real size.

Uncle Bob’s idea of a “foreman” could work under some circumstances. Like Big Design Up Front, however, it doesn’t scale. Collaboration is as important to the team leader as it is to the architect. The consequences of an all-knowing, all-powerful personality can be just as dire in this role as for an architect.

In “Hordes of Novices”, Bob Martin observed “When it’s possible to get a degree in computer science without writing any code, the quality of the graduates is questionable at best”. The problem here is that universities are geared to educate, not train. Just because training is more useful to an employer (at least in the short term), does not make education unimportant. Training deals with this tool at this time while how to determine which tool is right for a given situation is more in the province of education. It’s the difference between how to do versus how to figure out how to do. Both are necessary.

As I’ve already noted, it’s a thorny issue. Rather than offering an answer, I’d rather offer the opportunity for others to add to the conversation in the comments below. How did we get here and how do we go forward?