Tag Archives: Conway’s Law

Microservices or Monoliths – Fences and Neighbors

Posted on by

Photo of fence separating fields from a road

 

At the end of my last post, “What Makes a Monolith Monolithic?”, I stated that I didn’t consider the term “monolithic” to be inherently derogatory. It is, rather, a descriptive term relating to the style of organizing an application’s architecture. Depending on the context the system operates within, a monolithic architectural style could lie anywhere on the continuum between perfectly suited and perfectly disastrous. Placing it on that continuum requires a sense of what qualities are most needed or desired and which can be traded off in their stead. Everything comes with a cost, and attempting to ignore that fact merely sets us up for unpleasant future surprises.

After an initial period of unbridled enthusiasm, opinion seemed to gel around the idea that highly distributed application architectures (aka microservice architectures) were not suitable to all contexts. There are prerequisites for jumping into the microservices pool in terms of problem architecture, infrastructure, and organization. Attempting to shoehorn a microservice architecture into an environment that cannot support it will be overly expensive at best and a failure of apocalyptic proportions at worst.

There are many aspects of application design that are commonly recognized as beneficial: modularity, loose-coupling, high cohesion, and separation of concerns. It is critical to realize that these aspects can be found in systems with microservice architectures, monolithic systems, and everything in between. Distributed architectures are not necessary for modularity, nor any of those other aspects. In fact, one could easily create an application with a microservice architecture whose qualities are opposite to these desirable ones.

There are, however, situations where the benefit of a microservice architecture outweighs the costs and complexity. The ability to independently deploy and scale the various parts of an application is a major benefit, in my opinion. A well designed microservice architecture can even allow for the components of an application to be replaced on the fly. These features are not unique to microservice architectures, but are arguably easier to achieve than in other application architectures.

Real design, balancing both costs and benefits, is required. Sticking a bit of network in between the components is insufficient to ensure success. Deliberate design, especially as the boundaries multiply, is critical for an effective system. Identifying and providing for those boundaries at the conceptual level (i.e. before they become physical) is key. Good fences can either make for good neighbors, or they can create a maze of barriers.

Advertisement

Form Follows Function on SPaMCast 442

Posted on by

SPaMCAST logo

A new month brings a new appearance on Tom Cagley’s Software Process and Measurement (SPaMCast) podcast.

This week’s episode, number 442, features Tom’s excellent essay on capability teams (highly recommended!), followed by a Form Follows Function installment based on my post “Systems of Social Systems and the Software Systems They Create”. Kim Pries bats cleanup with a Software Sensei column, “Software Quality and the Art of Skateboard Maintenance”.

In this episode, Tom and I continue our discussion on the organizations as system concept and how systems must fit into their context and ecosystem. In my previous posts on the subject, I took more of a top down approach. With this post, I flipped things around to a bottom up view. Understanding how the social and software systems interact (including the social system involved in creating/maintaining the software system) is critical to avoid throwing sand in the gears.

You can find all my SPaMCast episodes using under the SPaMCast Appearances category on this blog. Enjoy!

Microservices, Monoliths, and Modularity

Posted on by

Iceberg

 

There are very valid reasons for considering a microservice architecture (MSA) when building/evolving an application. In my opinion, however, forcing modularity isn’t one of those very valid reasons.

Just the other day, I saw tweet from Simon Brown saying this same thing:

I still like his comment from two years back: “I’ll keep saying this … if people can’t build monoliths properly, microservices won’t help”. I believe that if you’re having problems building a monolith properly, trying to use a distributed architecture to force modularity may actually cause harm.

MSAs, like any distributed application architecture, involve increased complexity and costs; table stakes, if you will. Like an iceberg, there’s both a lot more to it than just what’s showing above the waterline and a fair amount of hazard for the unwary. If a development team cannot or will not comply with design guidelines (e.g. modularity requirements), injecting additional complexity is probably not the solution you need.

Distributing an application makes it harder to accidentally entangle different concerns, but it doesn’t make it impossible:

I’d argue that making it harder to accidentally break modularity addresses neither of the groups I mentioned earlier: those that cannot or will not comply. It’s ironic, but those who fail to understand the need for modularity can be very creative in their “solutions”, regardless of the obstacles. Likewise, those who refuse to comply.

In short, distribution as a means of “ensuring” modularity fails the fitness for purpose test.

The situation becomes worse when you factor in the additional complexity inherent in a distributed system. Likewise, there’s the cost of the table stakes (infrastructure, process, staffing, etc.) mentioned above. Of course, having abandoned the principle of cause and effect, one could attempt some “creative” workarounds to avoid having to pay the price (in other words, adding more and more complexity).

When you introduce significant additional complexity (with all its attendant risk) with little chance of the technique actually achieving its goal, you’ve caused harm.

These concerns are not solely limited to the application architecture. Distributing the data architecture has the same limitations in terms of ensuring modularity and introduces additional complexity. Adding boundaries adds the need for governance. A disciplined, monolithic team can maintain modularity in a monolithic data architecture. Multiple separate teams trying to share a monolithic data architecture will either experience a crippling level of governance overhead or a complete breakdown in modularity.

MSAs can be useful when you need independently scalable and replaceable components. When you have multiple teams working on one logical application, they can also be appropriate as well. Using the technique when the cost outweighs the potential payoff, however, is a losing bet.

Systems of Social Systems and the Software Systems They Create

Posted on by

I’ve mentioned before that the idea of looking at organizations as systems is one that I’ve been focusing on for quite a while now. From a top-down perspective, this makes sense – an organization is a system that works better when it’s component parts (both machine and human) intentionally work together.

It also works from the bottom up. For example, from a purely technical perspective, we have a system:

Generic System

However, without considering those who use the system, we have limited picture of the context the system operates within. The better we understand that context, the better we can shape the system to fit the context, otherwise we risk the square peg in a round hole situation:

Generic System with Users

Of course, the users who own the system are also only a part of the context. We have to consider the customers as well:

Generic System with Users and customers

Likewise, we need to consider that the customers of some systems can be internal to the organization while others are external. Some of the “customers” may not even be human. For that matter, sometimes the customer’s interface might be a human (user) rather than software. Things get complicated when we begin adding in the social systems:

Generic EITA with Users and customers

The situation is even more complicated than what’s seen above. We need to account for the team developing and operating the automated system:

Generic System with Users, customers, and IT team

And if that team is not a unified whole, then the picture gets a whole lot more interesting:

Generic System with Users, customers, and IT teams

Zoomed out to the enterprise level, that’s a lot of social systems. When multiplied by the number of automated systems involved, the number easily becomes staggering. What’s even more sobering is reflecting on whether those interactions have been intentionally structured or have grown organically over time. The interrelationship of social and software systems is under-appreciated. A series of tweets from Gregory Brown last week makes the same case:

A number of questions come to mind:

  • Is anyone aware of all the systems (social and software) in play?
  • Is anyone aware of all the interactions between these systems?
  • Are the relationships and interactions a result of intentional design or have they “just happened”?
  • Are you comfortable with the answers to the first three questions above?

Monolithic Applications and Enterprise Gravel

Posted on by

Pebbles

It’s been almost a year since I’ve written anything about microservices, and while a lot has been said on that subject, it’s one I still monitor to see what new pops up. The opening of a blog post that I read last week caught my attention:

Coined by Melvin Conway in 1968, Conway’s Law states: “Any organization that designs a system will produce a design whose structure is a copy of the organization’s communication structure.” In software development terms, Conway’s Law suggests that a given team will build apps that mirror the team’s organizational structure. Siloed functional teams produce siloed application architectures.

The result is a monolith: A massive application whose functionality is crammed into a few crowded parts. Scaling a simple pattern to the enterprise level often results in a monolith.

None of this is wrong, per se, but in reading it, one could come to a wrong conclusion. Siloed functional teams (particularly where the culture of the organization encourages siloed business units) produce siloed application architectures that are most likely monoliths. From an enterprise IT architecture aspect, though, the result is not monolithic. Googling the definition of “monolithic”, we get this:

mon·o·lith·ic
ˌmänəˈliTHik/
adjective
  1. formed of a single large block of stone.
  2. (of an organization or system) large, powerful, and intractably indivisible and uniform.
    “rejecting any move toward a monolithic European superstate”
    synonyms: inflexible, rigid, unbending, unchanging, fossilized
    “a monolithic organization”

Rather than “a single large block of stone”, we get gravel. The architecture of the enterprise’s IT isn’t “large, powerful, and intractably indivisible and uniform”. It may well be large, but its power in relation to its size will be lacking. Too much effort is wasted reinventing wheels and maintaining redundant data (most likely with no real sense of which set of data is authoritative). Likewise, while “intractably indivisible” isn’t a virtue, being intractable while also lacking cohesion is worse. Such an IT architecture is a foundation built on shifting sand. Lastly, whether the EITA is uniform or not (and I would give good odds that it’s not), is irrelevant given the other negative aspects. Under the circumstances, worrying about uniformity would be like worrying about whether the superstructure of the Titanic had a fresh paint job.

Does this mean that microservices are the answer to having an effective EITA? Hardly.

There are prerequisites for being able to support a microservice architecture; table stakes, if you will. However, the service-oriented mindset can be of value whether it’s applied as far down as the intra-application level (i.e. microservices – it is an application architecture pattern) or inter-application (the more traditional SOA). Where the line is drawn depends on the context of the application(s) and their ecosystem. What can be afforded and supported are critical aspects of the equation at all levels.

What is necessary for an effective EITA is a full-stack approach. Governance and data architecture in particular are important aspects to consider. The goal is consistent, intentional alignment across all levels (enterprise, EITA, solution, and application), promoting a cohesive architecture throughout, not a top-down dictatorship.

Large edifices that last are built from smaller pieces that fit together on purpose.

Can you afford microservices?

Posted on by

Check

Much has been written about the potential benefits of designing applications using microservices. A fair amount has also been written about the potential pitfalls. On this blog, there’s been a combination of both. As I noted in “Are Microservices the Next Big Thing?”: It’s not the technique itself that makes or breaks a design, it’s how applicable the technique is to problem at hand.

It’s important, however, to understand that “applicable to the problem at hand” isn’t strictly a technical question. The diagram in Philippe Kruchten‘s tweet below captures the full picture of a workable solution:

As Kruchten pointed out in his post ‘Three “-tures”: architecture, infrastructure, and team structure’, the architecture of the system, the system’s infrastructure, and the structure of the team developing the system are mutually supporting. These aspects of the architecture of the solution must be kept aligned in order for the solution to work. In my opinion, it should be a taken as a given that this architecture of the solution must also align with the architecture of the problem as a minimum condition to be considered fit for purpose.

Martin Fowler alluded to the need to align architecture, infrastructure, and team structure in “MicroservicePrerequisites” when he listed rapid provisioning, basic monitoring, and rapid deployment as pre-conditions for microservices. These capabilities not only represent infrastructure requirements, but also “…imply an important organizational shift – close collaboration between developers and operations: the DevOps culture”. Permanent product teams building and operating applications are, in my opinion, an extremely effective way to deliver IT. It must be realized, however, that effectiveness comes with a price tag, in terms of people, tools, and infrastructure.

In “MicroservicePremium”, Fowler further stated “don’t even consider microservices unless you have a system that’s too complex to manage as a monolith”, identifying “sheer size” as the biggest source of complexity. Size will encompass both technical and organizational concerns:

The microservice approach to division is different, splitting up into services organized around business capability. Such services take a broad-stack implementation of software for that business area, including user-interface, persistant storage, and any external collaborations. Consequently the teams are cross-functional, including the full range of skills required for the development: user-experience, database, and project management.

Expanding on this, the ideal organization will be one cross-functional team per microservice/bounded context. Even with very small teams, this requires either significant expenditure or a compromise of how the architectural and social aspects (i.e. Conway’s Law) work together in this architectural style.

Other requirements inherent in a microservice architecture are things like API governance and infrastructure services to support distributed processing (e.g. a service registry). Data considerations that are trivial in monolithic environment like transactions, referential integrity, and complex queries are absent in a distributed environment and facilities may need to be bought or built to compensate. In a distributed environment, even error logging requires special consideration to avoid drowning in complexity:

The overhead in terms of organization, infrastructure, and tooling, whether in ideal or comprised form, will introduce complexity and cost. I would, in fact, expect compromises to avoid costs to introduce even more complexity. If the profile of the system in terms of business value and necessary complexity (i.e. complexity inherent in the business function) warrants the additional overhead, then that overhead can represent a valid solution to the problem at hand. If, however, the complexity is solely created by the overhead, without an underlying need, the solution becomes suspect. Adding cost and complexity without offsetting benefits will likely lead to problems. Matching the solution to the problem and balancing those costs and benefits requires the attention of an architectural role at the application level, rather than relying on each team to work independently and hope for coherence and economy.

Full Stack Enterprises (Who Needs Architects?)

Posted on by

In my last post, “Locking Down the Prisoners: Control, Conflict and Compliance for Organizations”, I returned to a topic that I’ve been touching on periodically over the last year, organizations as systems, which overlaps significantly with the topic of enterprise architecture (not to be confused with enterprise IT architecture of which EA is a superset). While I do not pretend to be an expert on the subject, the fractal nature of the environment I work within as a software architect makes it difficult (perhaps even dangerous) to ignore.

Systems reside within ecosystems, which being systems themselves, reside within an ecosystem of their own. Both systems and ecosystems are mutually influencing and that influence must be understood to the extent possible and accounted for, both upstream and down. Where this fails to happen, coherence between information systems and social systems breaks down. Significantly, issues in systems at higher levels of granularity can negate positive aspects of the systems that comprise them.

This is probably a good place to point out that my use of terms like “social system” is not meant to remove the human aspect. On the contrary, social systems are intensely human in nature. Where those systems are dysfunctional, it is individuals that ultimately pay the price. The intent, is to point out the inter-relationships that make up our environment.

Consider the hypothetical systems outlined in “Making and Taming Monoliths”. Assuming all of the systems involved were modular enough and technically capable of interoperation, it would not be enough. Considerations of data architecture (starting with, which system is authoritative?) could spike the effort, or at least seriously delay it. Organizational structure (aka Conway’s Law), behavior (management, governance, process), and psychology/culture could all either impel or impede. A perfect example of this concept is the near requirement that DevOps be in place where microservices are used – it may be possible to develop and deploy microservices in other process models, but it will involve far more difficulty.

“Alignment” is a term that is often mentioned in terms of IT. However, even if an organization’s IT systems are perfectly aligned with the business units they support, it will do little good if those units are working at cross purposes. Just as the full stack of an application requires coherent, intentional design to work well, so too does an enterprise. This does not, however, imply that a rigid, top-down mode of operation is needed; it’s actually quite the opposite.

In “Auftragstaktik and fingerspitzengefühl”, Tom Graves described the UK’s World War II air defense system as an example of an effective enterprise. The two German words refer to the concepts of situational awareness (fingerspitzengefühl) and mission tactics (Auftragstaktik), which underlay the success of the system. In essence, situational awareness at each level was enabled by the flow of information up and down the hierarchy, while decisions appropriate to each level were made at that level informed by understanding of the situation and the commander’s intent. This is not just a matter of process. As Tom noted:

Another key element is organisational-culture – whether the culture invites or dissuades individual judgement within real-time action (auftragstaktik), elicitation and capture of real-world subtleties (for fingerspitzengefühl) and/or whistleblower-type algedonic responses.

An organization must not only be structured so as to achieve its aims, the drive to do so must be part of its DNA. Even when some components of the organization may have remits that conflict with others (think audit, accounting, InfoSec, etc.), the ultimate direction of all parts should harmonize. Structuring a system so as to resolve conflicts is an architectural practice, regardless of the type of system.

“Microservices, SOA, and EITA: Where To Draw the Line? Why to Draw the Line?” on Iasa Global

Posted on by

In my part of the world, it’s not uncommon for people to say that someone wouldn’t recognize something if it “bit them in the [rude rump reference]”. For many organizations, that seems to be the explanation for the state of their enterprise IT architecture. For while we might claim to understand terms like “design”, “encapsulation”, and “separation of concerns”, the facts on the ground fail to show it. Just as software systems can degenerate into a “Big Ball of Mud”, so too can the systems of systems comprising our enterprise IT architectures. If we look at organizations as the systems they are, it should be obvious that this same entropy can infect the organization as well.

See the full post on the Iasa Global site (a re-post, originally published here).

Form Follows Function on SPaMCast 347

Posted on by

SPaMCAST logo

This week’s episode of Tom Cagley’s Software Process and Measurement (SPaMCast) podcast features Tom’s essay on project management in an Agile environment (aka “Project Management is Dead”) and a Software Sensei column on testing from Kim Pries in addition to a Form Follows Function installment on microservices, Devops and Conway’s Law.

In SPaMCast 347, Tom and I discuss my “Fixing IT – Microservices and DevOps to the Rescue?” post, specifically on how microservice architectures are not just a technical approach but an organizational one as well.

Form Follows Function on SPaMCast 339

Posted on by

SPaMCAST logo

This week’s episode of Tom Cagley’s Software Process and Measurement (SPaMCast) podcast features Tom’s essay on demonstrations and a Form Follows Function installment on microservices, SOA, and Enterprise IT Architecture.

For SPaMCast 339, Tom and I discuss my “Microservices, SOA, and EITA: Where To Draw the Line? Why to Draw the Line?” post.