How well-managed are your dependencies? Are you aware of all of them? Which ones can fail gracefully? Which ones allow the application to continue in a degraded state in the event of a failure? How many dependencies would cause your application to become unavailable in the event of a failure?
See the full post on the Iasa Global Blog (a re-post, originally published here).
In a previous post, I quoted a passage from Ruth Malan’s “Conway’s Law”:
The organizational divides are going to drive the true seams in the system.
The architecture of the system gets cemented in the forms of the teams that develop it. The system decomposition drives team ownership. Then the organizational lines of communication become reflected in the interfaces, with cleaner, better preserved interfaces along the lines where organizational distance increases. In small, co-located teams, short-cuts can be taken to optimize within the team. But each short-cut that introduces a dependency is like rebar in concrete–structurally efficient, but rigid. If the environment changes, demanding new lines to be drawn, the cost becomes clear. The architecture is hard to adapt.
Those eight sentences contain extremely important considerations for application architecture, particularly if you are considering a microservice architecture.
Microservice architectures, although they have gained currency due to a recent post by James Lewis and Martin Fowler, are not a new concept. Lewis and Fowler note a pedigree going back as far as 2011, and in the opinion of Steve Jones, “Microservices is SOA, for those who know what SOA is”. Roger Sessions published a 3 part series in the fall of 2012 detailing his concept of a “Snowman Architecture” (Part 1, Part 2, Part 3), which is fundamentally similar to the Lewis and Fowler model.
According to Lewis and Fowler, the characteristics of a microservice architecture are:
* Note: I consider these to more in the realm of process than architectural style. That being said, I also consider most of these to be good practice for teams using this particular architectural style.
Composing a system of independent, autonomous components, as opposed to a monolith, allows them to be partitioned by business capability as well as be developed and maintained by a permanent team. This transforms Conway’s Law from an observation about the nature of software systems into a principle that can be used to improve the structure of those systems (“The organizational divides are going to drive the true seams in the system.”). The social system architecture will influence the software system architecture in any event, the question is whether that influence will be intentional or not.
The microservice architectural style has been described as “applying many of the principles of SOLID at an architectural level”. As opposed to monolithic architectures, it enables a high degree of flexibility, cohesion and composability (when done well). It is not, however, a free lunch. Distributed applications are more complex than monolithic ones in terms of development, operations, and governance. Accessing components that are out-of-process, not to mention over the network, is not the same as dealing with in-process components. Network latency becomes more and more important as applications become more distributed. Asynchronous operations can ease some of that pain, but at the cost increasing the complexity of the system.
Overly granular services can be another pitfall with this architectural style. In “Services, Microservices, Nanoservices – oh my!”, Arnon Rotem-Gal-Oz discusses the nanoservice anti-pattern where “…overhead (communications, maintenance, and so on) outweighs its utility”. To expand on his “so on”, security is one of those overhead aspects that can become more burdensome as the application becomes more distributed. At the very minimum, authorization, if the application is entirely internal, is more complicated than it would be with a monolith. External users and/or dependencies increase the scope of that security overhead, potentially adding multiple methods of authentication, encryption, etc.
Data architecture complexity is another consideration. It should be borne in mind that each microservice is its own self-contained system with its own data store(s). This adds fragmentation and most likely, redundancy, to the enterprise’s data architecture. Redundancy is not bad per se (e.g. caching), but it does require more in terms of governance to identify what source is authoritative and what others are not. Likewise, fragmentation is not always poor design (think bounded contexts with shared concepts), but it may require more effort to consolidate all the data for a given entity for reporting purposes. One might be tempted to “cheat” and go the shared database route, but that would be a mistake.
In a post on application boundaries, Martin Fowler observed that “essentially applications are social constructions…drawn primarily by human inter-relationships and politics rather than technical and functional considerations”. Realizing that, we can use the organizational dynamics to shape the components of our systems of systems in a way that helps, rather than harms the technical and functional considerations. The critical things to keep in mind are balance and intent.
In his recent post “No one wants SOA”, Kevin Orbaker identifies one of the issues with infrastructure in general and Service-Oriented Architecture (SOA) in particular:
SOA is like plumbing in a home/office. No one thinks about plumbing. No one cares about plumbing. They just want water to come out of the faucet when they turn the handle. The faucet is like the UI of your application. It is the final output from which the user interacts with. They see it. They touch it. They interact with it. They don’t know where the water comes from. They don’t really care. They just expect it to work.
So how do we get people interested in (interested enough to pay for) what they don’t care about?
See the full post on the Iasa Global Blog (a re-post, originally written for the old Iasa blog).
Last fall I had the pleasure of engaging Jeff Sussna, Christian Verstraete and Christian Reilly in a multi-day discussion on Twitter regarding IT’s relationship with cloud providers. As it developed, the conversation morphed into a discussion of potential metaphors for IT.
At the mention of “metaphor”, I’d imagine that eyes are glazing over, but bear with me while I establish its relevance to IT. Now Merriam-Webster tells us that “metaphor” is “a figure of speech in which a word or phrase literally denoting one kind of object or idea is used in place of another to suggest a likeness or analogy between them”. We find this same type of analogy used with patterns, the concept of which came out of Christopher Alexander’s A Pattern Language. In essence, these metaphors are patterns, communicating a potential solution to problems of enterprise IT architecture in the same way that “bridge” and “factory” communicate potential solutions to software design problems. As such, they provide a frame of reference to those of us within IT as well as those we serve. Their importance lies in the message they convey to both sides of the table.
Two candidate metaphors emerged from the discussion. Christian Reilly took the view that an IT department’s role should become like that of a concierge, a service aggregator. While agreeing in principle, I took the position that a better analogy was that of a general contractor. A general contractor works to meet the client’s needs, performing some services while subcontracting others out, in a manner similar to the concierge. The difference is that the contractor possesses a set of domain expertise useful for advising the client. The contractor is also bound by building codes that constrain him/her in delivering the customer’s requests. This describes what, in my opinion, the role of IT should be.
The dysfunctional situation described in the first post of this series, “Fixing IT – How to Make a Monster (Customer)”, defines the antithesis of an effective IT operation. “Fixing IT – Taming the (Customer) Beast”, the second post in the series, discusses some of the mindset changes needed to repair the relationship between IT and the business and why that relationship is so important. The third post, “Fixing IT – Products, Not Projects Revisited”, outlines the way delivery processes can affect IT’s relationship with the business. “Fixing IT – The Importance of Ownership”, the fourth post in the series, takes on how the issues of budget and ownership can affect the relationship. This post, will wrap up the series by bringing those issues together to outline my opinion on how IT can move forward in spite of the challenges it faces today.
The ZDNet post that inspired the discussion I mentioned above, “4 ways IT can rise above outside cloud competitors”, is an excellent example of how IT can turn things around by providing additional value to cloud services. This can be done by acting as a pseudo-VAR in place of acting as a competitor as the ZDNet article proposed. In either case, IT provides real value to the business via its specific knowledge of both the business domain and the in-house technical environment. This approach, advocated by Christian Verstraete in his HP Blogs post “Respond to Shadow-IT, source services from external providers”, is becoming more and more necessary because 7 out of 10 organizations are using unapproved cloud applications (according to a study from January, 2013).
IT has been in this position before. Thirty years ago the disruptive, security challenging technology that IT had no time and budget for was the PC. History teaches that that genie is not going back in the bottle. Instead, the winning approach is to enable the use of technology that furthers the goals of the business (which is all the business wants, anyway) in a safe manner.
The words “in a safe manner” above is what makes the general contractor metaphor appropriate. IT serves two sets of masters, the individual business units and the enterprise as a whole. Sometimes, the interests of a business unit can conflict with those of the organization as a whole. In these situations, IT has a valid reason to refuse to break the rules (although finding a way to achieve the desired outcome and stay within the rules is better than a flat refusal). What is important is that the rules, as such, are not owned by IT, but by the organization. IT is well placed to advise the enterprise about the technical capabilities and trade-offs around decisions regarding security, business processes, etc. It is inappropriate, however, for IT to attempt to dictate these decisions (this is actually an example of a business unit’s interests potentially conflicting with those of the enterprise). Steve Jones illustrated how counter-productive this usurpation can be in his post “Why your IT strategy failed or why the business hates IT”:
IT Guy: “The new systems have new processes and we need to tell you what they are so you can change.”
Business Guy: “Have you done an impact analysis against our current processes?”
IT Guy: “No we’ve just defined the Best Practice To-Be processes, you need to do the impact and change management. We need the meeting so we can tell you what the To-Be processes are”
Business Guy in a voice so dripping with sarcasm I thought we’d have a flood: “I look forward to our IT department telling the business what Best Practice is for our industry.”
IT Guy, completely failing to read the sarcasm: “Great we’ll get it organised”
Some take exception to referring to “the business”, observing that IT is part of the business. This, of course, is true. However, the best way to demonstrate that is not via the terms used in conversations, but via the support given to executing the enterprise’s strategy. Making things happen is the way show that support.
Architectural design as a form of storytelling is an established theme.
See the full post on the Iasa Global Blog (a re-post, originally written for the old Iasa blog).
Organizations that have a centralized budgeting and provision of IT services risk creating a disconnect between those that need technology services and those that provide those services. The first post in this series alluded to the problem – those needing the services have too little control over obtaining those services. If there is only a single provider with finite capacity and a finite budget, then there is the potential for consumers to be left out when demand exceeds supply. Governance processes set up to manage the allocation of services may be ineffective and can lead to dysfunctional behavior when business unit leaders attempt to game the system. Lastly, this situation creates a perverse incentive to gold plate projects in order to compensate for the uncertainty over when the business unit might next get the attention of IT.
The DevOps approach I suggested in my previous post only partially addresses the issues. Having a dedicated team means little if the size, composition, processes, and priorities of that team are outside your control. Much is heard about the concept of “alignment”, but mere alignment is insufficient. As Brenda Michelson noted in “Beware the “Alignment Trap””:
I started to use the term ‘business-IT integration’, because I’m thinking beyond traditional business-IT alignment. Alignment refers to the review and reconciliation of independent activities, in this context the reconciliation of business strategy and plans with IT strategy, architecture and plans.
For business to reap the true value of IT, business and IT must collaborate on the development of strategy, architecture and plans. This collaboration, which should continue through delivery and operations, is business-IT integration.
To achieve the integration Michelson referred to, one should bear in mind Conway’s law: “organizations which design systems … are constrained to produce designs which are copies of the communication structures of these organizations”. The reason to do so is elegantly expressed in Ruth Malan’s paraphrase of it: ” if the architecture of the system and the architecture of the organization are at odds, the architecture of the organization wins”. She goes on to say:
The organizational divides are going to drive the true seams in the system.
The architecture of the system gets cemented in the forms of the teams that develop it. The system decomposition drives team ownership. Then the organizational lines of communication become reflected in the interfaces, with cleaner, better preserved interfaces along the lines where organizational distance increases. In small, co-located teams, short-cuts can be taken to optimize within the team. But each short-cut that introduces a dependency is like rebar in concrete–structurally efficient, but rigid. If the environment changes, demanding new lines to be drawn, the cost becomes clear. The architecture is hard to adapt.
Enterprises are systems, social systems, but systems nonetheless. In the case of business units and IT, you have a social system creating, maintaining, and evolving digital systems. Assuming Conway’s law is true, simpler organizational structures should yield simplified systems (in both the technical and organizational realms). The opposite is also true, poor business architecture can actually reinforce dysfunction and prevent progress. Tom Graves post “Financial-architecture and enterprise-architecture” graphically illustrates just such a case where the accounting structure and approval processes conspired to prevent cost savings:
In fact, the business-case was obvious – or at least, obvious to anyone who’d actually looked at what we’d done. The potential savings in licence-fees alone ran into many millions of dollars a year – let alone the more subtle savings from re-using idle servers or freeing-up unneeded real-estate. By comparison, the costs were tiny: in some cases not much more than a few hours of paperwork, and overall perhaps a few staff-FTE for a few months at most.
But the sting in the tail was that requirement for proof, in terms of the organisation’s standard company-accounts. What we discovered, to our horror, was that the accounts themselves prevented us from doing so: the way those accounts were structured actually made it impossible to make almost any business-case at all for ever shutting anything down.
It should be clear that business structures and processes should be designed to facilitate the execution of business strategy. Part of this is putting technology decision-making into the hands of the business. IT is well positioned to inform those decisions in terms of technical trade-offs, but hardly qualified to decide from a business standpoint. IT Governance: How Top Performers Manage IT Decision Rights for Superior Results by Peter Weill and Jeanne W. Ross (excerpted by David Consulting Group – see table 5) shows that business-driven governance optimizes for profit and growth.
When making decisions about how to use technology in furtherance of business goals is coupled with the responsibility for funding those decisions, customers have greater incentive to make more prudent decisions about their use of technology. Additionally, when those costs are borne by the business unit, both the unit and the enterprise have a clearer picture of its financial state than if those IT costs are accounted for in IT’s budget.
When IT is in the position of advising on and enabling, rather than controlling, use of technology, some benefits accrue. When there’s no need to hide “Shadow IT” from the IT department, business units can take advantage of advice re: security, what other units are doing, etc. The same applies to using outsourcing to augment in-house IT. By being aware of these external capabilities, IT is positioned as a partner to the business unit in attaining its goals, allowing IT to play to its strengths. IT departments that can support business goals as advisers and coordinators of technology in addition to being providers can extend their capabilities with external providers without fear of replacement.
Designing applications that can also serve as components in an enterprise platform is doubtless more complex than designing a standalone one. The relevant question, however, is not whether complexity is added, but whether the complexity is necessary.
See the full post on the Iasa Global Blog (a re-post, originally written for the old Iasa blog).
Form Follows Function 