Systems of Social Systems and the Software Systems They Create

I’ve mentioned before that the idea of looking at organizations as systems is one that I’ve been focusing on for quite a while now. From a top-down perspective, this makes sense – an organization is a system that works better when it’s component parts (both machine and human) intentionally work together.

It also works from the bottom up. For example, from a purely technical perspective, we have a system:

Generic System

However, without considering those who use the system, we have limited picture of the context the system operates within. The better we understand that context, the better we can shape the system to fit the context, otherwise we risk the square peg in a round hole situation:

Generic System with Users

Of course, the users who own the system are also only a part of the context. We have to consider the customers as well:

Generic System with Users and customers

Likewise, we need to consider that the customers of some systems can be internal to the organization while others are external. Some of the “customers” may not even be human. For that matter, sometimes the customer’s interface might be a human (user) rather than software. Things get complicated when we begin adding in the social systems:

Generic EITA with Users and customers

The situation is even more complicated than what’s seen above. We need to account for the team developing and operating the automated system:

Generic System with Users, customers, and IT team

And if that team is not a unified whole, then the picture gets a whole lot more interesting:

Generic System with Users, customers, and IT teams

Zoomed out to the enterprise level, that’s a lot of social systems. When multiplied by the number of automated systems involved, the number easily becomes staggering. What’s even more sobering is reflecting on whether those interactions have been intentionally structured or have grown organically over time. The interrelationship of social and software systems is under-appreciated. A series of tweets from Gregory Brown last week makes the same case:

A number of questions come to mind:

  • Is anyone aware of all the systems (social and software) in play?
  • Is anyone aware of all the interactions between these systems?
  • Are the relationships and interactions a result of intentional design or have they “just happened”?
  • Are you comfortable with the answers to the first three questions above?

The Seductive Myth of Greenfield Development

Greger Wikstrand‘s tweet from earlier this week packed a wealth of inspiration into one image:

The second statement particularly resonated with me: “The present is built on the past.”

How often do we, or those around us, long for a chance to do things “from scratch”. The idea being, without the constraints of “legacy” code, we could do things “right”. While it’s a nice idea, it has no basis in reality.

Rewrites, of course, will involve dealing with existing data. I’ve yet to encounter a system where no one was interested in the data when it was replaced. I’ve shut down a few where there was no interest, but that’s a different story. The need for that existing data will serve as a potent influence on what can or cannot be done with the replacement system. Likewise, its structure. It’s not reasonable to assume that the data will be any less “legacy” than the code.

We might be tempted to believe that brand new systems escape this pitfall. In doing so, we fail to consider that new systems still must deal with the wants, needs, and attitudes of their stakeholders. People, processes, and organization form the ecosystem that new systems must fit into as surely as replacement systems must.

A crucial part of problem solving is having an adequate understanding of the problem. Everything has a backstory. Understanding the backstory is dependent on understanding the ecosystem the thing fits into. This what Sullivan was talking about when he said “…form ever follows function”.

Nothing’s Ex Nihilo.

Architecture in Context – Part 2

By Charlie Alfred and Gene Hughson

Up until this point, we’ve discussed what it means for Architecture to have context.  Contexts enable us to reason about the behavior of a group of stakeholders, whether they be buyers, end users, support staff, distributors, manufacturers or suppliers.

As we’ve pointed out, virtually all products or services are multi-context.  This means that the architecture of these products or services, if they expect to be effective, must also be multi-context.  A multi-context architecture is a lot like juggling.  You must balance your attention across an array of concerns and try to satisfy an array of stakeholders.  The following sidebar illustrates this point:

In the early 1960’s, the United States found itself engaged in a space race with the USSR.  Government officials in both countries  believed strongly that the first country to travel successfully in space and land men on the moon would have a significant military advantage.

President Kennedy collaborated with NASA to initiate a space program designed to put men on the moon’s surface and return them safely to earth, and achieve this by the end of the 1960’s.  On July 20, 1969, the Apollo 11 mission was successful at landing Neil Armstrong and Buzz Aldrin on the moon, and four days later, on July 24th, 1969, the crew safely splashed down in  the Pacific Ocean, where they were brought to safety by the USS Hornet.

The Apollo 11 mission is a high point of 20th Century US history for many.  It also does an excellent job of highlighting several concepts related to Architecture in Context.

  • Contexts:
    • U.S. Executive Branch,
    • U.S. Legislative Branch,
    • NASA astronauts
    • NASA scientists and engineers
    • U.S. Taxpayers, etc.
  • Value Expectations:
    • Land astronauts safely on the moon during 1960’s
    • Return the astronauts to earth safely
  • Pain Points:
    • Ceding space to the USSR would put the US Military at a strategic disadvantage
  • Priorities:
    • US President had more clout in this mission than Legislative Branch and NASA
    • Return the astronauts safely to earth is higher priority than landing on the moon
    • Landing on the moon is more important for NASA and US Executive Branch than for US Taxpayers as a whole
  • Challenges:
    • Difficulty of lunar landing complicated by moon’s craters and hills
    • Difficulty of space capsule reentry is driven by speed, atmospheric friction, and earth’s orbit (i.e.land in water, close enough to battleship)
    • Size and design of the space capsule depends on the power of the booster rockets that are needed to launch the capsule beyond the earth’s gravitational pull

[http://en.wikipedia.org/wiki/List_of_Apollo_missions]

Due to space constraints (pun intended), this sidebar merely scratches the surface of the interesting aspects of the Apollo 11 mission.  Another critical point is that the Apollo and Gemini missions that preceded Apollo 11 demonstrated feasibility of technical approaches to address important challenges.  For example, Apollo 7 was the first manned mission to orbit the earth and splashdown safely, and Apollo 8 was the first manned mission to orbit the moon and return safely.  It is worth noting how NASA recognized how difficult and important the reentry challenge was, and how early mission decisions were made to overcome it.

Balancing importance, difficulty and centrality (cross-dependency among challenges) can be a daunting problem.  What best practices exist to help you solve it?

Best Practice 1:  Identify and understand your key contexts.

Understanding your contexts means identifying them on the basis of similar goals, priorities, external forces, and pain points.  When doing this, be sure to focus on  the “why” questions:

  • Why do stakeholder groups A and B have the same pain point?
  • Do they have the same priority for relieving this pain point, or is this a much higher priority for one group vs. the other?
  • Is the pain point caused by the same external forces?
  • Is the pain point obstructing the same goals, or different goals?

Questions like these will enable you to cleanse your context notions so that the behaviors and external forces in each context are as consistent as possible.

Best Practice 2:  Understand your key contexts as soon as possible

One tendency in multi-context system development is to leave future contexts as some other day’s problem.  The tendency is to focus on the stakeholders for the first release as the only ones who matter:

  • We won’t be marketing to those groups (countries) for years, why waste time thinking about them now?
  • Those stakeholders are so new, they won’t know what they need.
  • Why spend all the money and  effort interviewing or doing other forms of research when their needs are likely to change?

These objections are based on the perception that identifying and understanding a context requires a lot of effort.  In reality, it requires much less than it appears.  A context is a generalization of behavior and the quality of this generalization can vary with how imminently it is needed.

I realize this sounds like it violates the YAGNI (“you ain’t going to need it”) principle, but it doesn’t.  There is a big difference between anticipating a future context and doing a little defensive design to accommodate its variations than there is of building in full support for it.  The first case is one of defining good interfaces to encapsulate variation.  The other is an implementation and testing burden.   Additionally, it is better to be aware of potential conflicts between contexts before resolving them involves significant design and code changes.

Best Practice 3:  Understand the challenges in satisfying each context’s pain points

Pain points and challenges are similar, and may be confused.  Challenges are concerns of the solution provider.  A challenge is one or more forces that must be overcome to provide value.
A pain point is similar, however it is linked to stakeholders within a context.  This difference shows up in goals, priorities and external factors are considered important.  For example, a pain point might be the need to keep trade secrets confidential from hackers, while the challenge might deal with specific types of cyber threats.

As the solution architect, we derive challenges from pain points in contexts.   Contexts are important here, as similar pain points may exist in related contexts, but with different goals, priorities, or even external challenges.  For example, within an investment management firm, response time needs differ for portfolio managers, traders and compliance officers.

Challenges are framed as problem statements — specific issues that the solution provider must overcome in order to provide value.  It is important to keep the relationships between contexts, pain points and challenges.  In general, contexts have many-to-many relationships to contexts and to pain points.  In other words, there could be several challenges for certain pain points.  Some challenges may be quite similar across several pain points, potentially spanning many contexts.

Best Practice 4:  Carefully consider the nature of challenges when combining over contexts

Challenges are like chemical elements.  Each has its own structure and properties.  However, in a system (especially a multi-context system), challenges combine and form molecules.  Molecules have their own chemical properties, aside from their constituent elements.  For example, Carbon and Oxygen independently are staples of life, while their combination, Carbon Monoxide, is an odorless, tasteless, and most notably, deadly gas.

As mentioned above, challenges take a common form:

  • How does the challenge create or detract from value expectations in a context?
  • Which external forces cause the challenge’s impact to be magnified or compounded

These two properties make it relatively easy to look at a challenge as a chemical element, or combine it with other challenges to view it as a  molecule.  Here are a few ways to characterize challenges that can be useful in examining their impact:

Some useful dimensions for categorizing challenges are:

Compatibility (of two challenges)

  • Compatible – independent problems, no issues solving simultaneously
  • Friction – partially dependent problems, some tradeoffs and/or risks
  • Antagonistic – highly dependent, serious tradeoffs and risks

Breadth (of a challenge)

  • Pervasive – scope of challenges reaches throughout solution
  • Regional – scope of challenges pervades an area but encapsulated
  • Local – scope of challenges limited to a narrow area

Occurrence (of a challenge)

  • Persistent – challenge occurs all or most of the time
  • Intermittent – challenge occurs with a predictable frequency
  • Conditional – challenge occurs in certain situations or conditions

These dimensions and categories can be useful for determining how to manage or aggregate challenges:

  • Within a context, as in how the challenge interacts and combines with others
  • Across context, as in whether similar challenges in two or more contexts are sufficiently alike to merge, or whether they combine into a more significant one

Best Practice 5: Prioritize challenges to identify the most important ones to address
 
Software architecture, just as software engineering, is a discipline of deciding among alternative approaches to reach a decision.  While some decisions get made in small sets (e.g. this system will be 3-Tier, .NET, IIS with a SqlServer database), many, if not most, decisions are made independently.

One important thing to remember is that virtually every decision made reduces the degrees of freedom for the rest.  Sometimes this reduction is desirable, but many times it paints the subsequent decisions into a corner.

If the goal of a software architect is to make good decisions for the system, it makes sense that s/he should address the highest priority challenges first (while the degrees of freedom are more available).  But how should challenges be prioritized?  Three criteria need to be balanced:

  1. Importance:
    • How many contexts does the challenge affect?
    • How much of an  impact does the challenge make on each one?
    • What is the weighted average of this impact, given the relative importance of each context?
  2. Difficulty:
    • The more difficult a challenge, the more degrees of freedom it is likely to need
    • The more difficult a challenge, the more degrees of freedom it will consume
    • The more degrees a challenge will consume, the earlier it should be tackled.
  3. Centrality (also called Core):
    • Challenges are also dependent on the solutions to other challenges
    • The dependent challenge should be addressed before or concurrently with the challenges that depend on them

Architectural design takes place in an environment of constraints.  Constraint should be understood as a neutral concept, because it both prevents and enables.  The same cup that constrains the flow of water also enables you to bring that water to your lips.  Managed appropriately, constraints provide the structure and form that yield the desired function(s).  Part of that management is understanding that design decisions are constraints.  Decisions made in isolation risk inappropriately constraining a design in terms of the whole.

Remediation of architectural constraints is, by its very nature, expensive.  Rewiring is more involved than repainting; replacing a foundation is far more extensive still.  Understanding and accounting for all of the contexts involved allows you to see the architecture of the problem as a whole.  The architecture of the problem then becomes the skeleton upon which the architecture of the solution can be built, incrementally, iteratively, and most important, effectively.

Architecture in Context – Part 1

By Charlie Alfred and Gene Hughson

It’s a common occurrence on online forums to see someone ask what architectural style is the right one. Likewise, it’s common to see a reply of “it depends” because “context is king”. Many will nod sagely at this; of course it is, context is king. But, what is context? Specifically, what do we mean by the term “context” in terms of software and solution architecture?

At a high level, context defines the problem environment for which a solution is intended to provide value. That definition, however, is far too superficial to suffice, because only the most trivial of systems will have a single context. Refining our definition, we can state that a context is a grouping of stakeholders sharing similar goals, priorities, and perceptions. In practice, unless you’re dealing with a system that you are developing yourself for your sole use, you will be dealing with multiple contexts, and they are likely to overlap like the Olympic logo. If you are puzzled by this statement, consider silverware. We need knives, forks and spoons and we have different types of each of them.

Identifying contexts and understanding how they interrelate gives definition to the architecture of the problem, which is a necessary prerequisite to designing the architecture of a solution (note the use of “a solution” instead of “the solution” – it’s intentional). This involves considerable work to identify the goals, priorities, and perceptions in that the natural tendency is for a stakeholder to ask for what they think will solve their problem rather than enumerate their problems (which is the point of the apocryphal “faster horses” quote frequently attributed to Henry Ford). Nonetheless, breaking out those raw needs is crucial to success.

Where you have multiple contexts, chances are the goals, priorities, and perceptions diverge rather than complement. Take, for example, a pickup truck. One context may be those who haul bulky and/or heavy goods daily. Another context may be made up of those who frequently tow trailers. A third context might be those who haul or tow from time to time, but mainly use the truck for transportation. The value of any given feature of the truck: miles per gallon of fuel, size of the bed, whether or not the bed is covered, quality of the sound system, torque, etc. will vary based on the context we’re considering. Additionally, external forces (such as the laws of physics) will come into play and complicate making design decisions. For example, long, open truck beds maximize hauling capacity, but will also harm fuel efficiency.

Another complicating factor is the presence of peripheral stakeholders, those whose contexts will affect the direct stakeholders. In the sense of our pickup truck example, we can consider mechanics as an example of this. Trucks that have more room in the engine compartment are easier (therefore cheaper) to service but will suffer in terms of fuel efficiency due to increased size and weight. In the IT realm, development and support teams would fall into this category. Although their contexts would not be primary drivers, ignoring them could well impose a cost on the direct stakeholders in terms of turnaround time for enhancements and fixes.

Examples of multi-context software systems can be found many places. The more generic a system, the more contexts it will have. Commodity office software (word processors, spreadsheets, etc.) are a prime example. Consider Microsoft Excel, which is a simple row/column tabulator, a statistical analysis tool, a database client, and, with its macro language, is an application development platform. Stretching to cover this many needs turns the Excel User Experience into a game of Twister. Platforms, such as Android are another example. Highly configurable applications like Salesforce.com (which has arguably crossed the line between product and platform) span multiple contexts as well. In the enterprise IT space, first SOA and now microservice architectures are nothing if not an attempt to address the plethora of contexts in play via separation of concerns.

A holistic consideration of the contexts at hand is an important success factor when evolving a system iteratively and incrementally. Without that consideration, decisions can be made that are conducive to one context but antagonistic to another. Deferring the reconciliation of the two contextual conflicts until later may result in considerable architectural refactoring. This refactoring will involve time and expense at a minimum, and if time is constrained, the likelihood of incurring technical debt is high. Each additional conflict that is blundered into due to the lack of up front consideration increases the risk to the system. This is not to endorse a detailed Big Design Up Front (BDUF), but a recognition that problem awareness and problem avoidance is cheaper than rework. If the context has been identified, then it’s YAGNI – not “You Ain’t Gonna Need It” but “You Are Gonna Need It”.

The next post will in this series will address concrete practices to integrate multiple contexts into a unified architecture of the problem that can serve as the foundation for a coherent architecture of a solution.

Product Owner or Live Customer?

Wooden Doll

Who calls the shots for a given application? Who gets to decide which features go on the roadmap (and when) and which are left by the curb? Is there one universal answer?

Many of my recent posts have dealt in some way with the correlation between organizational structure and application/solution architecture. While this is largely to be expected due to Conway’s Law, as I noted in “Making and Taming Monoliths”, many business processes depend on other business processes. While a given system may track a particular related set of business capabilities, concepts belonging to other capabilities and features related to maintaining them, tend to creep in. This tends to multiply the number of business stakeholders for any given system and that’s before taking into account the various IT groups that will also have an interest in the system.

Scrum has the role of Product Owner, whose job it is to represent the customer. Alan Holub, in a blog post for Dr. Dobb’s, terms this “flawed”, preferring Extreme Programming’s (XP) requirement for an on-site customer. For Holub, introducing an intermediary introduces risk because the product owner is not a true customer:

Agile processes use incremental design, done bit by bit as the product evolves. There is no up-front requirements gathering, but you still need the answers you would have gleaned in that up-front process. The question comes up as you’re working. The customer sitting next to you gives you the answers. Same questions, same answers, different time.

Put simply, agile processes replace up-front interviews with ongoing interviews. Without an on-site customer, there are no interviews, and you end up with no design.

For Holub, the product owner introduces either guesses or delay. Worst of all in his opinion is when the product owner is “…just the mouthpiece of a separate product-development group (or a CEO) who thinks that they know more about what the customer wants than the actual customer…”. For Holub, the cost of hiring an on-site customer is negligible when you factor in the risks and costs of wrong product decisions.

Hiring a customer, however, pretty much presumes an ISV environment. For in-house corporate work, the customer is already on board. However, in both cases, two issues exist: one is that a customer that is the decision-maker risks giving the product a very narrow appeal; the second is that the longer the “customer” is away from being a customer, the currency of their knowledge of the domain diminishes. Put together, you wind up with a product that reflects one person’s outdated opinion of how to solve an issue.

Narrowness of opinion should be sufficient to give pause. As Jim Bird observed in “How Product Ownership works in the Real World”:

There are too many functional and operational and technical details for one person to understand and manage, too many important decisions for one person to make, too many problems to solve, too many questions and loose ends to follow-up on, and not enough time. It requires too many different perspectives: business needs and business risks, technical dependencies and technical risks, user experience, supportability, compliance and security constraints, cost factors. And there are too many stakeholders who need to be consulted and managed, especially in a big, live business-critical system.

As is the case with everything related to software development, there is a balance to be met. Coherence of vision enhances design to a point, but then can lead to a too-insular focus. Mega-monoliths that attempt to deal comprehensively with every aspect of an enterprise, if they ever launch, are unwieldy to maintain. By the same token, business dependencies become system dependencies and cannot be eliminated, only re-arranged (microservices being a classic example – the dependencies are not removed, only their location has changed). The idea that we can have only one voice directing development is, in my opinion, just too simplistic.