Making and Taming Monoliths

In Meditation XVII of Devotions upon Emergent Occasions, John Donne wrote the immortal line “No man is an island, entire of itself; every man is a piece of the continent, a part of the main.” If we skip ahead nearly 400 years and change our focus from mortality to information systems (lots of parallels there), we might re-word it like this: No system is an island, entire of itself; every system is a piece of the enterprise, a part of the main.

In “Carving it up – Microservices, Monoliths, & Conway’s Law”, I discussed how the new/old idea of microservice architectures related to the more monolithic classic style of application architecture. An important take-away was the applicability of Conway’s Law to application and solution architectures. Monoliths are monolithic for a reason; a great many business activities are dependent on other business activities performed by actors from other business units. In Ruth Malan’s words: “The organizational divides are going to drive the true seams in the system.” Rather than a miscellaneous collection of business capabilities, monoliths typically represent related capabilities stitched together in one application.

The problem with monoliths is that there is a tendency for capabilities/data to be duplicated across multiple systems. The dominant concern of one system (e.g. products in an inventory system) will typically be a subordinate concern in another (e.g. products in an order fulfillment system). This leads to different units using different systems based on their needs vis-a-vis a particular concern. Without adequate governance, this leads to potential conflicts over which system is authoritative regarding data about those concerns.

Consider the following diagram of a hypothetical family of systems consisting of an order management system and a system to track the outsourced activities of the subset of orders that are fulfilled by vendors. Both use miscellaneous data (such as the list of US states and counties) that may or may not be managed by a separate system. They also use and maintain duplicate from other systems. The order system does this with data from the pricing, customer management, and product management systems. It also exports data for manual import into the accounts receivable system. The outsourced fulfillment system does this with data from the product management and vendor management systems as well as the order system. It exports data for manual import into the accounts payable system. Because of the lack of integration, there is redundant data with no clear ability to declare any one set as the master one. Additionally, there is redundant code as the functions of maintaining this data are repeated (and likely repeated in an inconsistent manner) across multiple systems.

There are different responses to the issue of uncontrolled data duplication, ranging from “hope for the best” (not recommended) to various database replication schemes. Jeppe Cramon’s “Microservices: It’s not (only) the size that matters, it’s (also) how you use them – part 4” outlines the advantages of publishing data as events from authoritative source systems for use by consumer systems. This provides near real-time updates and avoids the pitfall of trying to treat services like a database (hint, latency will kill you). Caching this data locally should not only improve performance but also allows for the consumer applications to manage attributes of those concerns that are unique to them with referential integrity (if desired).

In the diagram below, the architecture of this family of systems has been rationalized by the introduction of different types of services appropriate to the different use cases. The order system now makes use of the pricing system solely as a service (with its very narrow focus, the pricing system is definitely amenable to the microservice architectural style) and no longer contains data or code related to that concern and only contains enough code to trigger asynchronous messages to the accounts receivable system. Likewise, the outsourced fulfillment system contains only a thin shell for the accounts payable system. Other concerns (customers, vendors, products, etc.) are now managed by their respective owing systems and updates are published asynchronously as events. The mechanism for distributing these events (the cloud element in the diagram) is purposely not named as there are multiple ways to accomplish this (ESBs are one mechanism, but far from the only one). While the consuming systems cache the data locally, much of the code that previously was required to maintain them is no longer needed. In many cases, entire swathes of “administration” UI can be done away with. Where necessary, it can be pared down to only edit data related to those concerns that is both unique to and restricted to the consuming application.

I will take the time to re-emphasize the point that not all communication need be, nor should it be, the same. Synchronous communication (shown as a solid line in the diagram) may be appropriate for some usages (particularly where an end user is waiting on an immediate success/fail type of response). An event-driven style (shown as a dotted line in the diagram) will be much more performant and scalable for many more situations. As I noted in “Coordinating Microservices – Playing Well with Others”, designing and coding distributed applications in the same manner as a monolith will result in a web of service dependencies, a distributed big-ball of mud. Coupling that is a code smell in a monolith can become a crippling issue in distributed systems.

Disassembling monoliths in this manner is complex. It can be done incrementally, removing the need for expensive “big bang” initiatives. There is, however, a need for strategy and governance at the right level of detail (micromanagement and too-granular design are still unlikely to yield the best results). Done well, you can wind up with both better data integrity and less code to maintain.