Pragmatic Application Architecture

I saw a tweet on Friday about a SlideShare deck that looked interesting, so I bookmarked it to read later. As I was reading it this morning, I found myself agreeing with the points being made. When I got to the next to the last slide, I found myself (or at least, this blog) listed alongside some very distinguished company under “Reading Material”.

Thanks, Bart Blommaerts and nice job!

Designing Communication, Communicating Design

The Simplest metamodel in the world ever!

We work in a communications industry.

We create and maintain systems to move information around in order to get things done. That information moves between people and systems in combinations and configurations too numerous to count. In spite of that, we don’t do that great a job of communicating what should be, for us, extremely important information. We tend to be really bad at communicating the architecture of our systems – structure, behavior, and most importantly, the reasons for the decisions made. It’s bad enough when we fail to adequately communicate that information to others, it’s really bad when we fail to communicate it to ourselves. I know I’ve let myself down more than once (“What was I thinking here?!”).

Over the past few days, I’ve been privileged to follow (and even contribute a bit to) a set of conversations on Twitter. Grady Booch, Ivar Jacobson, Ruth Malan, Simon Brown, and others have been discussing the need for architectural awareness and the state of communicating architecture.

This exchange between Simon, Chris Carroll and Eoin Woods sums it up well:

First and foremost, an understanding of what the role of a software architect is and why it’s important is needed. Any organization where the role is seen as either just a senior developer or (heaven help us!) some sort of Taylorist “thinker” who designs everything for the “worker bee” coders to implement, is almost guaranteed to be challenged in terms of application architecture. Resting on that foundation of shifting sand, the organization’s enterprise IT architecture (EITA) is likewise almost guaranteed to be challenged barring a remarkable series of “happy accidents”. The role (not necessarily position) of software architect is required, because software architecture is a distinct set of concerns that can either be addressed intentionally or left to emerge haphazardly out of the construction of the system.

Before we can communicate the architecture of a system, it’s necessary to understand what that is. In “Software Architecture: Central Concerns, Key Decisions”, Ruth Malan and Dana Bredemeyer defined it as high impact, systemic decisions involving (at a minimum):

  • system priority setting
  • system decomposition and composition
  • system properties, especially cross-cutting concerns
  • system fit to context
  • system integrity

I don’t think it’s possible to over-emphasize the use of “system” and “systemic” in the preceding paragraphs. That being said, it’s important to understand that architectural concerns do not exist in a void. There is a cyclic relationship between the architectural concerns of a system and the system’s code. The architectural concerns guide the implementation, while the implementation defines the current state of the architecture and constrains the evolution of future state of the architecture. Code is a necessary, but insufficient source of architectural knowledge – it’s not enough. As Ruth Malan noted in the Visual Design portion (part II) of her presentation at the Software Architect Conference in London a year ago:

Slide from Ruth Malan's presentation on Visual Design

While the code serves as a foundation of the system, it’s also important to realize that the system exists within a larger context. There is a fractal set of systems within systems within ecosystems. Ruth illustrated this in the Intention and Reflection portion (part III) of the presentation reference above:

Slide from Ruth Malan's presentation on Intention and Reflection

[Note: Take the time to view the entirety of the Intention and Reflection presentation. It’s an excellent overview of how to design the architecture of a system.]

The fractal nature of systems within systems within ecosystems is illustrated by the image at the top of the post (h/t to Ric Phillips for the reblog of it). Richard Sage‘s humorous (though only partly, I’m sure) suggestion of it as a meta-model goes a long way towards portraying the problem of a language to communicate architecture.

Not only are we dealing with a nested set of “things”, but the understanding of those things differ according to the stakeholder. For example, while the business owner might see a “web site” as one monolithic thing, the architect might see an application made up of code components depending on other applications and services running on a collection of servers. Maintaining a coherent, normalized object model of the system yet being able to present it in multiple ways (some of which might be difficult to relate) is not a trivial exercise.

Lower-level aspects of design lend themselves to automated solutions, which can increase reliability of the model by avoiding “documentation rot”. An interesting (in my opinion) aspect that can also be automated is the evolution of code over time. What can’t be parsed from the code, however, is intention and reasoning.

Another barrier to communication is the need to be both expressive and flexible (also well illustrated by Richard’s meta-model) while also being simple enough to use. UML works well on the former, but (rightly or wrongly) is perceived to fail on the latter. Simon Brown’s C4 model aims to achieve a better balance in that aspect.

At present, I don’t think we have one tool that does it all. I suspect that even with a suite of tools, that narrative documents will still be way some aspects are captured and communicated. Having a centralized store for the non-code bits (with a way to relate them back to the code) would be a great thing.

All in all, it is encouraging to see people talking about the need for architectural design and the need to communicate the aspects of that design.

Leadership Anti-Patterns – The Great Pretender

Roman Mosaic with Tragedy & Comedy Masks as gargoyles above a water basin

My previous leadership type, the Growler, was hard to classify as it had aspects of both pattern and anti-pattern. The Great Pretender, however, is much easier to label. It’s clearly an anti-pattern.

Before entering the working world full-time, I worked in the retail grocery business (both of my parents also had considerable industry experience, both retail and wholesale). I ran into this type more than once. The type is distinguished by a lack of domain knowledge and/or experience, coupled with an apparent inability to trust anyone with knowledge and/or experience. Consequently, the default method of decision-making appeared to be “whatever someone suggests, do something different”. It was as if someone had mixed impostor syndrome and the Dunning-Kruger effect together and skimmed off the most detrimental parts of each.

I remember one July Fourth holiday where a Great Pretender tripled the order for sliced-bread and cut the order for hamburger and hotdog rolls in half. This was based on reading something that said Americans were eating healthier. Unfortunately, that message wasn’t communicated to the Americans in our community (bear in mind, this was many years ago and July Fourth) and we wound up with customers unhappy that we had run out of what they wanted to buy and a lot of soon-to-expire bread that had to be marked down drastically so that it sold before going out of date. The failure to trust subordinates with the right expertise carries costs.

Another Great Pretender questioned his stock crew when he found them taking a break while a truckload of merchandise remained in the back room. The response, “Do you know how long it takes to get all that put on the shelf?” sent him scurrying away. The crew, who were malingering, had a good laugh.

The combination of lack of knowledge and lack of trust opens the door to an interesting manipulation strategy. When you want something from a Great Pretender, you never ask for it directly. It’s always “Boss, should I do this incredibly inane thing that no one in their right mind would do or should I do what I actually want to do?” The response from the Great Pretender is always “Do that second thing” (every single time). I leave it to you, dear Reader, to only use this knowledge for good, not evil.

The idea that someone in a leadership position should be the best at all they oversee is a pretty common one. More than once I’ve seen people claim that the have no respect for a leader that can’t do their job better than them. This attitude, however, fundamentally misunderstands the nature of leadership (hint, effective leadership is more about coordinating the team than doing any one job on the team). This attitude also demonstrates a lack of understanding of the cognitive capacity that would be required to lead a team involved in a minimally complicated undertaking (hint, effective leadership is more about coordinating the team than doing any one job on the team). This attitude also ignores the fact that a leader is responsible for tasks unique to their position (hint, effective leadership is more about coordinating the team than doing any one job on the team). When a team member has this attitude, it can be a problem.

When the leader buys into this attitude, we get the Great Pretender.

Leaders have their own roles and responsibilities to fulfill. This involves dealing with what’s appropriate to their role and relying on others for what’s appropriate to theirs. This requires communication and collaboration. Micro-managing and insecurity are counter-productive. The best leaders, in my opinion, are those that can recognize talent in others and gather around themselves a team of people with complementary strengths. They’re not the experts, but expert at helping a collection of experts come together for a common purpose. That involves placing trust in those being led.

Having to know it all can be fatal.

“Distance…is the one true enemy…”

Gregory Brown tweeted a great series on the problem of distance last week:

It’s amazing how much information can be conveyed in nine tweets. It’s amazing how many aspects of a very complex socio-technical undertaking, software development, are affected by this concept of distance. I would argue that this concept of distance applies likewise to the social systems that software development belongs to as a component part.

Distance between action and result as well as distance between result and response are just as much a problem for social systems as software systems. This was one of the main themes of “The Ignorance of Management – Deep and Wide”, trying to manage too far down the hierarchy just doesn’t scale. There are too many decisions at too deep a level of detail across too many areas for this to be effective.

It’s a case of mismatched impedance, resulting in overload. Increased distance equals increased transmission time, meaning that remote decisions will either take longer (risking timeliness of the decision) or will have to be made with less consideration (risking the fitness of the decision). Likewise, the greater number of decisions being made due to an inappropriate distance will force the same set of trade-offs. Time spent on lower-level issues also reduces time available for issues that are appropriate to the decision-maker’s level. This places even more pressure on the decision-maker in terms of being either hasty or late.

Ironically, better control is likely to come from delegating decision-making to the appropriate level of the organization than attempting to micro-manage. The true test of leadership, in my opinion, is not how things run when a leader is present, but how things run when they’re not. Adding distance stacks the deck, and not in your favor.

Form Follows Function on SPaMCast 399

SPaMCAST logo

This week’s episode of Tom Cagley’s Software Process and Measurement (SPaMCast) podcast, number 399, features Tom’s essay “Storytelling: Developing The Big Picture for Agile Efforts”, Kim Pries on deliberate practice, and a Form Follows Function installment on customer-centricity for IT.

Tom and I discuss my post “A Meaningful Manifesto for IT”. It seems obvious that the business of IT is meeting needs, but how many organizations are really happy with what they’re getting? The prevalence of “shadow IT” would seem to indicate that there’s some real discontent.

You can find all my SPaMCast episodes using under the SPAMCast Appearances category on this blog. Enjoy!

Dealing with Technical Debt Like We Mean it

What’s the biggest problem with technical debt?

In my opinion, the biggest problem is that it works. Just like the electrical outlet pictured above, systems with technical debt get the job done, even when there’s a hidden surprise or two waiting to make life interesting for us at some later date. If it flat-out failed, getting it fixed would be far easier. Making the argument to spend time (money) changing something that “works” can be difficult.

Failing to make the argument, however, is not the answer:

Brenda Michelson‘s observation is half the battle. The argument for paying down technical debt needs to be made in business-relevant terms (cost, risk, customer impact, etc.). We need more focus on the “debt” part and remember “technical” is just a qualifier:

The other half of the battle is communicating, in the same business-relevant manner, the costs and/or risks involved when taking on technical debt is considered:

Tracking what technical debt exists and managing the payoff (or write off, removing failed experiments is a reduction technique) is important. Likewise, managing the assumption of technical debt is critical to avoid being swamped by it.

Of course, one could take the approach that the only acceptable level of technical debt is zero. This is equivalent to saying “if we can’t have a perfect product, we won’t have a product”. That might be a difficult position to sell to those writing the checks.

Even if you could get an agreement for that position, reality will conspire to frustrate you. Entropy emerges. Even if the code is perfected and then left unchanged, the system can rot as its platform ages and the needs of the business change. When a system is actively maintained over time without an eye to maintaining a coherent, intentional architecture, then the situation becomes worse. In his post “Enterprise Modernization – The Next Big Thing!”, David Sprott noted:

The problem with modernization is that it is widely perceived as slow, very expensive and high risk because the core business legacy systems are hugely complex as a result of decades of tactical change projects that inevitably compromise any original architecture. But modernization activity must not be limited to the old, core systems; I observe all enterprises old and new, traditional and internet based delivering what I call “instant legacy” [Note 1] generally as outcomes of Agile projects that prioritize speed of delivery over compliance with a well-defined reference architecture that enables ongoing agility and continuous modernization.

Kellan Elliot-McCrea, in “Towards an understanding of technical debt”, captured the problem:

All code is technical debt. All code is, to varying degrees, an incorrect bet on what the future will look like.

This means that assessing and managing technical debt should be an ongoing activity with a responsible owner rather than a one-off event that “somebody” will take care of. The alternative is a bit like using a credit card at every opportunity and ignoring the statements until the repo-man is at the door.

The Ignorance of Management – Deep and Wide

Iceberg

While on LinkedIn a couple of weeks ago, an interesting graphic caught my eye. Titled “The Iceberg of Ignorance”, it referred to a 1989 study in which:

…consultant Sidney Yoshida concluded: “Only 4% of an organization’s front line problems are known by top management, 9% are known by middle management, 74% by supervisors and 100% by employees…”

The metaphor of the iceberg is simple to understand. The implications of these numbers, however, require some unpacking so as to understand the full nature of the problem. Once the problem is better defined, effective solutions can then be proposed.

A naive reading of this would be that the line-level employees know everything and that the higher you travel up the hierarchy, the more out of touch you get. That reading, however, fails on two counts. Firstly, it ignores the qualification of “front line”, which will not apply to all problems faced by the enterprise. Secondly, it fails to account for the fact that while 100% of front line problems will be known by front line employees, that’s not the same as saying that each front line employee will know 100% of front line problems.

It’s a question of cognitive capacity, both depth and width. As organizations grow, the idea that any one person could be aware of each and every detail of the operation becomes laughable, even assuming perfect communication (which is an extreme assumption). This difficulty is compounded by cultural conditioning to expect those in charge to know what’s going on:

Unfortunately, I suspect the vast majority of leaders and managers believe they should have all the answers — even though they couldn’t possibly know everything that’s going on at all levels and in all departments within their organization. And even though the world is changing so quickly that what we know right this second … may not be true and accurate anymore … in this second.

But because we’ve been entrained to have all the right answers, all the time, many of us put on a brave face and pretend we know — particularly when our boss asks us a question, or when a direct-report does. After all, we want to look good. We want to seem “on top of things.”

Pretending to have all the answers is stressful. It’s lonely. It’s draining.

And what if, when we are pretending to know, we give an answer that we later discover is wrong? Yikes! Now what?

In this situation, many people feel forced to “stick to their guns,” even in the face of conflicting evidence. So they wind up suffering from stress, anxiety and fear that they’ll be found out.
They may even hide the “correct answer” to save face, which certainly doesn’t do their conscience — or their company — any good.

Can you see how this need to have all the answers, all the time, can contribute to a culture of assumptions, half-truths and even outright lies?

In this sort of environment, do you think people are connecting deeply and sharing freely? Of course not. They’re competing with one another and hoarding information, because they believe the person with the right answer wins!

This culture of denial, delusion, and deception is how organizations arrive at the extreme situation I discussed in my last post, “Volkswagen and the Cost of Culture”. Casual dishonesty, towards others and yourself, leads to habitual dishonesty. Corruption breeds corruption. Often, it’s not even coldly calculated evil designed to profit, but ad hoc “going along to get along” to avoid consequences – impostor syndrome on an epic scale.

Command and control (the term of art from military science, not the pejorative description for micromanagement) is a subject with a long history. It’s frequently abbreviated as C3, since communication is an integral component of the discipline. A pair of techniques with a pedigree going back to the 18th century have a track record of success.

In his post “Auftragstaktik and fingerspitzengefühl”, Tom Graves describes these techniques:

The terms originate from the German military, from around the early-19thC and mid-20thC respectively. They would translate approximately as:

The crucial point here is to understand that they work as a dynamic pair, to provide a self-updating bridge between strategy, tactics and operations, or, more generally, between plan and action.

In the post, Tom describes these techniques through the example of the air defense system used by Britain in WWII. In this system, information flowed into the command centers from observers, and radar installations. The information was combined and contextualized, then sent back out to airfields and anti-aircraft batteries where it was used to repel air attacks. Tom noted that this is often depicted as a linear flow:

Yet describing the Dowding System in this way kinda misses the point – not least that there’s alot of information coming back from each of the front-line units at the end of that supposed one-way flow. Instead, the key here is that auftragstaktik and fingerspitzengefühl provide afeedback-loop that – unlike classic top-down command-and-control – is able to respond to fast-paced change right down to local level.

The linear-flow description also misses the point that it depends on more than information alone – there are key human elements without which the auftragstaktik / fingerspitzengefühl loop risk fading away into nothingness. For example, auftragstaktik is deeply dependent on trust, which in turn depends on a sense of personal connection and personal, mutual commitment, whilst fingerspitzengefühl depends on a more emotive form of sensing, of feeling, of an often-literal sense of ‘being in touch‘ with what’s going on out there in the real-world

The Dowding system worked by combining effective communication and realistic command & control methods. Bi-directional communication improved situational awareness both up and down the hierarchy, providing detail to the upper layers and context to the lower ones. Realistic command and control can be summarized like this: since no one can possibly have full breadth and depth of knowledge about the situation, provide direction appropriate to your level (in accordance with the directions you received) and delegate to your subordinates the decisions appropriate to their level. In theory, as well as largely in practice, this resulted in decisions being made by those with the best knowledge to do so (again, guided by general direction from above).

A system that works with reality, rather than against it? What a concept.