Making and Taming Monoliths

In Meditation XVII of Devotions upon Emergent Occasions, John Donne wrote the immortal line “No man is an island, entire of itself; every man is a piece of the continent, a part of the main.” If we skip ahead nearly 400 years and change our focus from mortality to information systems (lots of parallels there), we might re-word it like this: No system is an island, entire of itself; every system is a piece of the enterprise, a part of the main.

In “Carving it up – Microservices, Monoliths, & Conway’s Law”, I discussed how the new/old idea of microservice architectures related to the more monolithic classic style of application architecture. An important take-away was the applicability of Conway’s Law to application and solution architectures. Monoliths are monolithic for a reason; a great many business activities are dependent on other business activities performed by actors from other business units. In Ruth Malan’s words: “The organizational divides are going to drive the true seams in the system.” Rather than a miscellaneous collection of business capabilities, monoliths typically represent related capabilities stitched together in one application.

The problem with monoliths is that there is a tendency for capabilities/data to be duplicated across multiple systems. The dominant concern of one system (e.g. products in an inventory system) will typically be a subordinate concern in another (e.g. products in an order fulfillment system). This leads to different units using different systems based on their needs vis-a-vis a particular concern. Without adequate governance, this leads to potential conflicts over which system is authoritative regarding data about those concerns.

Consider the following diagram of a hypothetical family of systems consisting of an order management system and a system to track the outsourced activities of the subset of orders that are fulfilled by vendors. Both use miscellaneous data (such as the list of US states and counties) that may or may not be managed by a separate system. They also use and maintain duplicate from other systems. The order system does this with data from the pricing, customer management, and product management systems. It also exports data for manual import into the accounts receivable system. The outsourced fulfillment system does this with data from the product management and vendor management systems as well as the order system. It exports data for manual import into the accounts payable system. Because of the lack of integration, there is redundant data with no clear ability to declare any one set as the master one. Additionally, there is redundant code as the functions of maintaining this data are repeated (and likely repeated in an inconsistent manner) across multiple systems.

There are different responses to the issue of uncontrolled data duplication, ranging from “hope for the best” (not recommended) to various database replication schemes. Jeppe Cramon’s “Microservices: It’s not (only) the size that matters, it’s (also) how you use them – part 4” outlines the advantages of publishing data as events from authoritative source systems for use by consumer systems. This provides near real-time updates and avoids the pitfall of trying to treat services like a database (hint, latency will kill you). Caching this data locally should not only improve performance but also allows for the consumer applications to manage attributes of those concerns that are unique to them with referential integrity (if desired).

In the diagram below, the architecture of this family of systems has been rationalized by the introduction of different types of services appropriate to the different use cases. The order system now makes use of the pricing system solely as a service (with its very narrow focus, the pricing system is definitely amenable to the microservice architectural style) and no longer contains data or code related to that concern and only contains enough code to trigger asynchronous messages to the accounts receivable system. Likewise, the outsourced fulfillment system contains only a thin shell for the accounts payable system. Other concerns (customers, vendors, products, etc.) are now managed by their respective owing systems and updates are published asynchronously as events. The mechanism for distributing these events (the cloud element in the diagram) is purposely not named as there are multiple ways to accomplish this (ESBs are one mechanism, but far from the only one). While the consuming systems cache the data locally, much of the code that previously was required to maintain them is no longer needed. In many cases, entire swathes of “administration” UI can be done away with. Where necessary, it can be pared down to only edit data related to those concerns that is both unique to and restricted to the consuming application.

I will take the time to re-emphasize the point that not all communication need be, nor should it be, the same. Synchronous communication (shown as a solid line in the diagram) may be appropriate for some usages (particularly where an end user is waiting on an immediate success/fail type of response). An event-driven style (shown as a dotted line in the diagram) will be much more performant and scalable for many more situations. As I noted in “Coordinating Microservices – Playing Well with Others”, designing and coding distributed applications in the same manner as a monolith will result in a web of service dependencies, a distributed big-ball of mud. Coupling that is a code smell in a monolith can become a crippling issue in distributed systems.

Disassembling monoliths in this manner is complex. It can be done incrementally, removing the need for expensive “big bang” initiatives. There is, however, a need for strategy and governance at the right level of detail (micromanagement and too-granular design are still unlikely to yield the best results). Done well, you can wind up with both better data integrity and less code to maintain.

Advertisement

Fears for Tiers – Do You Need a Service Layer?

One of the perils of popularity (at least in technology) is that some people equate popularity with efficacy. Everyone’s using this everywhere, so it must be great for everything!

Actually, not so much.

No one, to my knowledge, has created the secret sauce that makes everything better with no trade-offs or side effects. Context still rules. Martin Fowler recently published “Microservices and the First Law of Distributed Objects”, which touches on this principle. In it, Fowler refers to what he calls the First Law of Distributed Object Design: “don’t distribute your objects”. To make a long story short, distributed objects are a very bad idea because object interfaces tend to be fine-grained and fine-grained interfaces perform poorly in a distributed environment. “Chunkier” interfaces are more appropriate to distributed scenarios due to the difference in context between in-process and out of process communication (particularly when out of process also crosses machine, even network boundaries). Additionally, distributed architectures introduce complexity, adding another contextual element to be accounted for when evaluating whether to use them.

In his post, under “Further Readings”, Fowler references an older post of his on Dr. Dobb’s titled “Errant Architectures” (a re-packaging of Chapter 7 of his book Patterns of Enterprise Application Architecture). In that, he discusses encountering what was a common architectural anti-pattern of the time (early 2000s):

There’s a recurring presentation I used to see two or three times a year during design reviews. Proudly, the system architect of a new OO system lays out his plan for a new distributed object system—let’s pretend it’s some kind of ordering system. He shows me a design that looks rather like “Architect’s Dream, Developer’s Nightmare” with separate remote objects for customers, orders, products and deliveries. Each one is a separate component that can be placed in a separate processing node.
I ask, “Why do you do this?”

“Performance, of course,” the architect replies, looking at me a little oddly. “We can run each component on a separate box. If one component gets too busy, we add extra boxes for it so we can load-balance our application.” The look is now curious, as if he wonders if I really know anything about real distributed object stuff at all.

The quintessential example of this style was the web site presentation layer with its business and data layer behind a set of web services. Particularly in the .Net world, web services were the cutting edge and everyone needed them. Of course the system architect in the story was confusing scalability for performance (as were most who jumped onto the web service layer bandwagon). In order to increase the former, a hit is taken to the latter. In most cases, this hit is entirely unnecessary. A web application with an in-process back-end can be load-balanced much more simply than separate UI and service sites, without suffering the performance penalty of remote communication.

There are valid use cases for a service layer, such as when an application provides a both an interactive user interface and is service-enabled. When there are multiple user interfaces, some front ends will benefit from physical separation from the back-end (native mobile apps, desktop clients, SharePoint web parts, etc.). Applications can be structured to accommodate the same back-end either in-process or out of process. This structure can be particularly important for applications that have both internal (where internal is defined as built and deployed simultaneously with the back-end) and external service clients as these have different needs in terms of versioning. Wrapping the back-end with a service layer rather than allowing it to invade a particular service implementation can allow one back-end to serve internal and external customers either directly or via services (RESTful and/or SOAP) without violating the DRY principle.

Many of the same caveats that apply to composite applications formed from microservices (network latency, serialization costs, etc.) apply to these types of applications. It makes little sense to take on these downsides in the absence of a clear demonstrable need. Doing so looks more like falling prey to a fad than creating a well thought out design.

[Wedding Cake Image by shine oa via Wikimedia Commons.]