Babies, Bathwater, and Software Architects

'Fixing Problems' - XKCD 1739

I try to be disciplined about my writing (picking themes, creating a backlog, collecting notes and links on those topics, etc.), but it seems like serendipity won’t be denied, no matter what I do. On the same day that XKCD published this cartoon, Erik Dietrich published “Software Architect as a Developer Pension Plan”. While I agree with a lot that Erik says in the post, I think he also gets into a “throwing out the baby with the bathwater” situation by dismissing the need for the software architect role, part of which the cartoon illustrates.

Before I go any farther, I do need to point out that I’ve been following and interacting with Erik for about four years now. It’s a friendly disagreement; meant to be more debate than argument.

In the post, Erik makes the point (which I largely agree with) that companies see the role of software architect through a Taylorist lens. They see the software architect as a “thinker”, meant to drive a herd of “doers”:

The modern corporation, like Taylor before it, loosely divides into three categories of humans: owners/executives, managers, and grunts. Owners own and charge executives with executing their will. Executives delegate to managers. Managers think and assemble the workers into org charts designed to optimize efficiency. Workers keep their simple heads down and do as they’re told.

Theoretically, proponents would say, this applies to any domain and type of work. Corporations form themselves into these pyramids without considering any other options, whether they’re private military outfits, gigantic manufacturing facilities, or companies designing and selling software. Of course, the reality turns out to be that not all operations do well splitting thinking and labor. Let’s pick one at random. Oh, I dunno, say, writing software.

We try it. We hire junior developers to be directed by senior ones. And all of them fall in line under the watchful guise of a “tech lead,” who defers to a council of architects. And somewhere at the center of the organizational maelstrom stands The Lead Architect, directing elaborate bucket marches, water flows, and all sorts of magic, like Mickey Mouse in Fantasia. It’s a beautiful, cascading symphony of work flowing from the cleverest down to the simplest. Or… at least, that’s how it goes on paper.

Erik rightly argues that this model is fatally flawed. This is doubly so in the case of the example he gives, an individual offered a position as a software architect doing the thinking for a set of offshore “commodity” developers. Offshore development can be done well (that’s a post I’ll save for another day), but not using that model. Where Erik goes wrong, in my opinion, is in adopting this flawed definition of the software architect role.

Developers should be perfectly competent to do their own thinking. If they’re not, no software architect will be able to help. Even if that person were capable of handling the load of providing detailed directions for several developers, doing so would prevent them from attending to their own areas of concern. It would require being able to anticipate and make all the functional design decisions up front, as well as communicating them to multiple “typists” quicker than they could mindlessly hammer out the code, while still having time to consider all the requisite cross-cutting, quality of service considerations for an application. Do we really need to bother with experiments to determine that this is beyond improbable?

There is a separation between the developer role and software architect role, but it needs to be clearly understood as a difference in focus. A developer’s focus is more vertical, concentrating on implementing functionality while maintaining/improving/not harming the quality of service (aka the horrendously misnamed “non-functional”) aspects of the system. A software architect’s focus should be more horizontal, concentrating on the architecturally significant quality of service aspects that will enable the system to meet the needs of its stakeholders (or, at least, not unreasonably prevent the system from doing so).

The difference between the roles is a matter of thinking at different levels of abstraction, not of one being superior to the other. The two mindsets are both necessary and complementary. A high-quality architectural design without quality implementation cannot produce a high-quality system. Likewise, where there is implementation without architectural coordination, the quality of the resulting system is a matter of chance. For very small systems maintained by small, permanent teams, it may not be necessary for these roles to be distinct. In my experience, however, dealing with both the broadly general and the very specific at the same time scales poorly. Non-trivial systems quickly come to require architectural leadership, otherwise people are left fixing problems caused by problems fixed previously.

Leadership has been the main theme of almost all of my posts this month and has figured prominently in several of what’s been written this year. Many of these posts are assigned to the “Architectural Practice” category. This is because I absolutely believe that the software architect role is a leadership role. That being said, I don’t consider the Taylorist model to be effective leadership practice. In fact, I consider it an anti-pattern.

I’ve long been an advocate of a more collaborative model of practice. Rather than dictating, the software architect’s role should be one of consensus building and communication. Significant architectural disagreement indicates an issue with one or more members of the team. Significant uncertainty about the architecture indicates an issue with the software architect role.

Effective systems require both breadth and depth of effectiveness. This holds true for software systems as much as social systems. For the social systems producing software systems for use by other social systems (i.e. software development teams), it is imperative.

Leadership Anti-Patterns – The Thinker

'The Thinker' by Rodin

My interest in leadership, how it works and how it fails, goes back a long way. Almost as soon as I learned how to read, history, particularly military history, has been a favorite of mine. Captains and kings, their triumphs and their downfalls, fascinated me. The eleven years I served with the Henrico Sheriff’s Office honed that interest. It allowed me to go beyond theory and the hearsay attendant in just reading about leaders. It allowed me to both observe firsthand and to get practical experience of my own.

For most of the latter part of my career with the Sheriff’s Office, I led a unit responsible for providing support services for the jail (canteen, recreation, laundry, hair-care, maintenance, and coordination of volunteer programs). In addition to their primary function, members of my unit, as sworn staff, also served as a ready reserve to the security staff during the busiest parts of the day. It was a challenge for me, because I was responsible for managing a group of experts with diverse duties. It was a challenge for them, because the way I led was something of a change for them.

Like everyone, I started my career working a security platoon in the jail. My second posting, however, was in our training academy. Roughly half of my responsibilities involved record keeping and administration of the training program. The other half of my time was spent in class, delivering training. Something about that resonated with me; I loved it and it strongly influenced my leadership style.

When necessary, I was fully capable of issuing orders to be complied with immediately. The operative word here being “necessary”. What’s necessary under emergency conditions and what’s necessary the other ninety-nine percent of the two are two different things. The majority of the time, my leadership style involved more coaching and little or no dictating.

I still laugh when I think of the first time a subordinate came to me for a decision. After her laying out the problem, rather than serving up a solution, I asked a simple question, “What do you want to do about it?” The speechless expressions I received in response were priceless. Having someone in authority ask for her opinion was so utterly foreign as to be beyond belief.

[Note – this is not the same thing as “don’t bring me problems, bring me solutions”. That type of bumper sticker philosophy is a good way to have people avoid you when they don’t know what to do, which is when they actually need you the most (and likewise, when you most need to know what they’re dealing with).]

By making sure that those who worked under me knew the boundaries they had to work with, we were able to divide things into three groups: the routine that was unremarkable, that which they could deal with on their own and tell me later, and that which they needed to get me involved in up front. Even with the last category, I asked for opinions. If it was a decision that I could make, my bias was to go with their recommendation unless I had a really good reason not to (as I noted above, most were specialists and knew the details of their position far better than I, so overruling them solely because I was “in charge” would be a really bad idea). When the decisions were above my pay grade, I would still give them credit and endorse their recommendation. My goal was to have a unit that could operate as well when I was not there as when I was.

At this point, you may be wondering where’s the information about this “Thinker” anti-pattern. All the background above, is to serve as contrast to one of my colleagues, a lieutenant in charge of a jail security platoon. This person informed their platoon that they were paid to do the thinking and their subordinates were paid to do the doing. No one other than this lieutenant was allowed to make any decisions, period. While I doubt this person had ever heard of Taylorism, they had a firm grasp of the principles of it.

Under the most normal of circumstances, there’s much to criticize about The Thinker’s way of doing things. The method doesn’t scale. It places a lot of burden on the Thinker. Getting a bathroom break in between decisions to be made is hard enough, but vacation? Forget about it.

The thing about a jail is that it’s not the most normal of circumstances, so there’s an added disadvantage. In the event of a hostage situation (something that’s more of a risk in a correctional environment), if the Thinker was the hostage, then they would be in for a very long ordeal. Unless one of their subordinates decided to ignore the Thinker’s orders and call for help, they would be stuck until someone “authorized” to think reported for duty.

Ignorance of the law (of unintended consequences) is no excuse.

Leadership Anti-Patterns – The Great Pretender

Roman Mosaic with Tragedy & Comedy Masks as gargoyles above a water basin

My previous leadership type, the Growler, was hard to classify as it had aspects of both pattern and anti-pattern. The Great Pretender, however, is much easier to label. It’s clearly an anti-pattern.

Before entering the working world full-time, I worked in the retail grocery business (both of my parents also had considerable industry experience, both retail and wholesale). I ran into this type more than once. The type is distinguished by a lack of domain knowledge and/or experience, coupled with an apparent inability to trust anyone with knowledge and/or experience. Consequently, the default method of decision-making appeared to be “whatever someone suggests, do something different”. It was as if someone had mixed impostor syndrome and the Dunning-Kruger effect together and skimmed off the most detrimental parts of each.

I remember one July Fourth holiday where a Great Pretender tripled the order for sliced-bread and cut the order for hamburger and hotdog rolls in half. This was based on reading something that said Americans were eating healthier. Unfortunately, that message wasn’t communicated to the Americans in our community (bear in mind, this was many years ago and July Fourth) and we wound up with customers unhappy that we had run out of what they wanted to buy and a lot of soon-to-expire bread that had to be marked down drastically so that it sold before going out of date. The failure to trust subordinates with the right expertise carries costs.

Another Great Pretender questioned his stock crew when he found them taking a break while a truckload of merchandise remained in the back room. The response, “Do you know how long it takes to get all that put on the shelf?” sent him scurrying away. The crew, who were malingering, had a good laugh.

The combination of lack of knowledge and lack of trust opens the door to an interesting manipulation strategy. When you want something from a Great Pretender, you never ask for it directly. It’s always “Boss, should I do this incredibly inane thing that no one in their right mind would do or should I do what I actually want to do?” The response from the Great Pretender is always “Do that second thing” (every single time). I leave it to you, dear Reader, to only use this knowledge for good, not evil.

The idea that someone in a leadership position should be the best at all they oversee is a pretty common one. More than once I’ve seen people claim that the have no respect for a leader that can’t do their job better than them. This attitude, however, fundamentally misunderstands the nature of leadership (hint, effective leadership is more about coordinating the team than doing any one job on the team). This attitude also demonstrates a lack of understanding of the cognitive capacity that would be required to lead a team involved in a minimally complicated undertaking (hint, effective leadership is more about coordinating the team than doing any one job on the team). This attitude also ignores the fact that a leader is responsible for tasks unique to their position (hint, effective leadership is more about coordinating the team than doing any one job on the team). When a team member has this attitude, it can be a problem.

When the leader buys into this attitude, we get the Great Pretender.

Leaders have their own roles and responsibilities to fulfill. This involves dealing with what’s appropriate to their role and relying on others for what’s appropriate to theirs. This requires communication and collaboration. Micro-managing and insecurity are counter-productive. The best leaders, in my opinion, are those that can recognize talent in others and gather around themselves a team of people with complementary strengths. They’re not the experts, but expert at helping a collection of experts come together for a common purpose. That involves placing trust in those being led.

Having to know it all can be fatal.

Innovation on Tap

Beer Tap

Two articles from the same site (, both dealing with planned innovations, but with dramatically different results:

While the article about the Amazon leak doesn’t report on customer reactions, that response is unlikely to be negative for a variety of reasons, all of which involve benefit to the customer. The most important one, the big innovation (in my opinion), is that opening physical stores allows it to take advantage of a phenomenon that other retailers find problematic:

Another way Amazon eviscerates traditional retailers is via the practice of showrooming. That’s where you go to a brick-and-mortar store and find the product you want but then buy it on Amazon — sometimes standing right there in the store, using Amazon’s mobile app.

Physical Amazon stores can encourage and facilitate showrooming, because they will have friendly salespeople to help you find, install and use the app. Once they teach you how to showroom in an Amazon retail store, you can continue showrooming in other stores.

While large retailers have been able to “combat” showrooming by embracing it, selling items at online prices in physical stores digs deeply into profit margins. When your business model is predominantly brick and mortar, the hit is greater than when your model is predominantly online. In short, if they play their cards right, Amazon will be able to better serve their customers without hurting their own bottom line.

Awareness of your customers’ wants and needs is key to making decisions that are more like Amazon’s and less Twitter’s. An intentional, collaborative approach, such as that described by Greger Wikstrand in his post “Developing the ‘innovation habit’”, is one way to promote that awareness:

When I worked at Ericsson Research, I instigated a weekly one-hour innovation meeting for my team. ‘Can you really be innovative every Thursday at 9am?’ you may cynically ask. Well, actually, yes, you can.

What we did in that hour was commit to innovation, dedicating a place and a time to our individual and collective innovative mindsets. Sometimes this resulted in little ideas that helped us do things incrementally better. And sometimes—just sometimes—those innovation hours were the birthplace of big ideas.

Collaboration is important because “Architect knows best” is as much a design folly at the enterprise level as it is at the application level. A better model is that described by Tom Graves in “Auftragstaktik and fingerspitzengefühl”, where both information and guidance flow both up and down the hierarchy to inform decisions both strategic and tactical. These flows provides the context so necessary to making effective decisions.

You can’t pull a tap and draw a glass of innovation, but you can affect whether your system makes innovative ideas more or less likely to be recognized and acted on.

This is part eleven of a conversation Greger Wikstrand and I have been having about architecture, innovation, and organizations as systems. Previous posts in the series:

  1. “We Deliver Decisions (Who Needs Architects?)” – I discussed how the practice of software architecture involved decision-making. It combines analysis with the need for situational awareness to deal with the emergent factors and avoiding cognitive biases.
  2. “Serendipity with Woody Zuill” – Greger pointed me to a short video of him and Woody Zuill discussing serendipity in software development.
  3. “Fixing IT – Too Big to Succeed?” – Woody’s comments in the video re: the stifling effects of bureaucracy in IT inspired me to discuss the need for embedded IT to address those effects and to promote better customer-centricity than what’s normal for project-oriented IT shops.
  4. “Serendipity and successful innovation” – Greger’s post pointed out that structure is insufficient to promote innovation, organizations must be prepared to recognize and respond to opportunities and that innovation must be able to scale.
  5. “Inflection Points and the Ingredients of Innovation” – I expanded on Greger’s post, using WWI as an example of a time where innovation yielded uneven results because effective innovation requires technology, understanding of how to employ it, and an organizational structure that allows it to be used well.
  6. “Social innovation and tech go hand-in-hand” – Greger continued with the same theme, the social and technological aspects of innovation.
  7. “Organizations and Innovation – Swim or Die!” – I discussed the ongoing need of organizations to adapt to their changing contexts or risk “death”.
  8. “Innovation – Resistance is Futile” – Continuing on in the same vein, Greger points out that resistance to change is futile (though probably inevitable). He quotes a professor of his that asserted that you can’t change people or groups, thus you have to change the organization.
  9. “Changing Organizations Without Changing People” – I followed up on Greger’s post, agreeing that enterprise architectures must work “with the grain” of human nature and that culture is “walking the walk”, not just “talking the talk”.
  10. “Developing the ‘innovation habit’” – Greger talks about creating an intentional, collaborative innovation program.

Inflection Points and the Ingredients of Innovation

WWI Photo Montage

One of my hobbies is the study of history. Not the dry, dusty, “…on this date these people did that” type of history, rather I’m fascinated by the story of how real people interacted with each other and the world around them. I’m interested in the brilliance and the stupidity, the master strokes and the blunders, the nobility and the infamy; all of which are often found combined in the same person. I like the type of history that, in explaining the world of yesterday, is helpful in making sense of the world of today.

World War I is one of those large-scale human tragedies that holds lessons for today. Teetering on the edge of two very different times, it mixed the tools of industrial warfare (machine guns, armored vehicles, and air power) with infantry tactics barely different from those of a hundred years previous. Over a little more than four years, old and new collided cataclysmically. It provides a brutally stark picture of the uneven nature of innovation and how that uneven nature can yield both triumph and tragedy.

So, just what does this have to do with information technology, systems development, and software architecture?

Quite a lot, actually, if you look beyond the surface.

Over the last few weeks, I’ve been having a running conversation with Greger Wikstrand on this blog and Twitter, about various aspects of IT. “We Deliver Decisions (Who Needs Architects?)” wove together some observations about situational awareness and confirmation bias into a discussion about the need for and aim of the practice of architectural design. In response, Greger posted a link to a video with him and Woody Zuill on serendipity in software development. That prompted my last post, “Fixing IT – Too Big to Succeed?”, discussing how an embedded IT approach could be used to foster the organizational agility needed to provide IT both effectively and efficiently.

Greger responded with “Serendipity and successful innovation”, making a number of extremely important points. For example, while fortune does favor the prepared, recognition of an opportunity is not enough. Likewise, an organizational structure that doesn’t impede innovation is important, but not impeding innovation is not the same as actively driving it forward. Evaluation of ideas is critical as well, in order to differentiate “…between the true gold of serendipity and the fool’s gold of sheer coincidence”. That evaluation, however, needs to be nimble as well. He closed the post with an example of small-company innovation on the part of a large organization.

That last part was what brought WWI to mind. The first and second World Wars were very different affairs (both horrible, but in different ways). Quite a lot of the technologies that we think of as being characteristic of the second (e.g. warplanes and tanks), originated in the first. Having a technology and discovering how best to employ it are two very different things. Certainly there was some technical evolution over twenty years, but the biggest change was in how they were employed.

In some cases, the opposite problem was in play. The organizational structures used by the combatants had proved their effectiveness for over a hundred years. Mission tactics, AKA auftragstaktic (whereby a commander, instead of detailed instructions, provides a desired outcome to his subordinates who are then responsible for achieving that outcome within broad guidelines), was also well-known at the time, but while radio existed, it was not yet portable enough to link dispersed mobile front line units with headquarters. Instead, that communication was limited to field phones and runners, and tactical communication was limited by the range of the human voice, forcing units to bunch up.

Without all the ingredients (technology, the understanding of how to employ it, and an effective organizational structure) outcomes are likely to be poor. Information must flow both up and down the hierarchy, otherwise decisions are being made blindly and without coordination. The parts must not only perform their function, but must also collaborate together. Organizations that fail to operate effectively across the full stack fail to operate effectively at all.

Fixing IT – Too Big to Succeed?

Continuing our discussion that I mentioned in my last post, Greger Wikstrand tweeted the following:

I encourage you to watch the video, it’s short (7:39) and makes some important points, which I’ll touch on below.

Serendipity, when it occurs, is a beautiful thing. Serendipity can occur when heads-down order taking is replaced with collaboration. Awareness of business concerns on the part of IT staff enhances their ability to work together with end users to create effective solutions. Awareness, however, is insufficient.

Woody Zuill‘s early remarks about the bureaucracy that gets in the way of that serendipity struck a nerve. Awareness of how to address a need does little good if the nature of the organization either prevents the need from being addressed. Equally bad is when a modest need must be bloated beyond recognition in order to become “significant” enough for IT intervention. Neither situation works well for the end users, IT, or the business as a whole.

So what’s needed to remedy this?

In “Project lead time”, John Cook asserted that project lead team was related to company size:

A plausible guess is that project lead time would be proportional to the logarithm of the company size. If a company with n employees has a hierarchy with every manager having m subordinates, the number of management layers would be around logm(n). If every project has to be approved by every layer of management, lead time should be logarithmic in the company size. This implies huge companies only take a little longer to start projects than medium-sized companies, and that doesn’t match my experience.

While I agree in principle, I suspect that the dominant factor is not raw size, but the number of management layers Cook refers to. In my experience, equally sized organizations can be more or less nimble depending on the number of approvals and the reporting structure of the approvers. Centralized, shared resources tend to move slower than those that are federated and dedicated. Ownership increases both customer satisfaction and their sense of responsibility for outcomes. Thus, embedding IT staff in business units as integral members of the team seems a better choice than attempting to “align” a shifting set of workers temporarily assigned to a particular project.

The nature of project work, when combined with a traditional shared resource organization, stands as a stumbling block to effective, customer-centric IT. IT’s customers (whether internal, external, or both) are interested in products, not projects. For them, the product isn’t “done” until they’re no longer using it. While I won’t go so far as to say #NoProjects, I share many of the same opinions. Using projects and project management to evolve a product is one thing, I’ve found project-centric IT to be detrimental to all involved.

Having previously led embedded teams (which worked as pseudo-contractors: employed by IT, with the goals and gold coming from the business unit), I can provide at least anecdotal support to the idea that bringing IT and its customers closer pays dividends. Where IT plays general contractor, providing some services and sub-contracting other under the direction of those paying the bills, IT can put its expertise to use meeting needs by adding value rather than just acting as order-takers. This not only allows for the small wins that Woody Zuill mentioned, but also for integrating those small wins into the overall operation of the business units, reducing the number of disconnected data islands.

Obviously there are some constraints around this strategy. Business units need to be able to support their IT costs out of their own budget (which sounds more like a feature than a bug). Likewise, some aspects of IT are more global than local. The answer there, is matching the scope of the IT provider unit with the scope of the customer unit’s responsibility. Federating and embedding allows decisions to be made closer to the point of impact by those with the best visibility into the costs and benefits. Centralization forces everything upwards and away from those points.

Operating as a monolithic parallel organization that needs to be “aligned” to the rest of the organization sounds like a system that’s neither effective nor efficient, certainly not both. When the subject is the distance between the decision and its outcome, bigger isn’t better.

Who Needs Architects? Well, Nobody Needs this Kind

The question above came up while recording SPaMCast 357 with Tom Cagley, and it’s an extremely important one. The post we were discussing, “Who Needs Architects? Because YAGNI Doesn’t Scale”, is one of many discussing the need for architectural design in software development. While I’m firmly convinced that the need is real, it should also be realized that there is a real danger in unilaterally imposing the design on a team.

Tom’s question about an “aristocracy of architects” was taken from his post “Re-Read Saturday: The Mythical Man-Month, Part 4 Aristocracy, Democracy and System Design”, part of a series in which he is reviewing Frederick Brooks’ The Mythical Man-Month. In the essay reviewed in this post, “Aristocracy, Democracy and System Design”, Brooks discussed the importance and value of conceptual integrity (i.e. a cohesive, unified design) to software systems. While I agree wholeheartedly that both architectural design and someone (or more than one someone) responsible for that design is necessary, I disagree that establishing an aristocracy is beneficial or even necessary. In fact, the portion labeled “Reality” on the graphic in Kelly Abuelsaad‘s tweet below, although talking about imposter syndrome, also illustrates why dictating design can be a bad idea.

One can certainly influence, even control the architecture of a system via a mandate. The problem with being given control is that no one can give effectiveness to go with it. As such, it’s brittle, subject to the limitations of the person given the authority and the compliance of those implementing the system. This brittleness exists even when the architect stays within their level of detail. Combining a dictatorial style with Big Design Up Front all but ensures failure.

In my experience, a participative, collaborative style of design yields better designs. In addition to benefiting from a variety of skills and experience, it also engenders greater understanding and ownership across the team. Arrogance, on the other hand, can be costly.

I firmly believe that a product will benefit from having someone whose focus is the cross-cutting, architecturally significant concerns. I also believe that part of that job is teaching and mentoring as well as listening to the rest of the team so that architectural awareness permeates the entire team. There are many aspects to being an architect, but being a dictator should not be one of them.

Updated 4/8/2016 to fix a broken link.