Architecture Corner: No Money – Lights Out! – Seven Deadly Sins of IT

Episode 5 of this season of Architecture Corner is out (I made a guest appearance in episode 1, “Good at Innovation”). In this installment, Chris the CEO falls victim to yet another temptation.

The CEO has decided that no more time and money will be “wasted” on old systems. Can Joakim Lindbom convince him that sloth breeds technical debt, leading to expensive outages?

Building a Legacy

Greek Trireme image from Deutsches Museum, Munich, Germany


Over the last few weeks, I’ve run across a flurry of articles dealing with the issue of legacy systems used by the U.S. government.

An Associated Press story on the findings from the Government Accountability Office (GAO) issued in May reported that roughly three-fourths of the $80 billion IT budget was used to maintain legacy systems, some more than fifty years old and without an end of life date in sight. An article on about the same GAO report detailed seven of the oldest systems. Two were over 56 years old, two 53, one 51, one 35, and one 31. Four of the seven have plans to be replaced, but the two oldest have no replacement yet planned.

Cost was not the only issue, reliability is a problem as well. An article on noted:

Then there’s the fact that, up until 2010, the Secret Service’s computer systems were only operational about 60% of the time, thanks to a highly outdated 1980s mainframe. When Senator Joe Lieberman spoke out on the issue back in 2010, he claimed that, in comparison, “industry and government standards are around 98 percent generally.” It’s alright though, protecting the president and vice president is a job that’s really only important about 60 percent of the time, right?

It would be easy to write this off as just another example of public-sector inefficiency, but you can find these same issues in the private sector as well. Inertia can, and does, affect systems belonging to government agencies and business alike. Even a perfectly designed implemented system (we’ve all got those, right?) is subject to platform rot if ignored. Ironically, our organizations seem designed to do just that by being project-centric.

In philosophy, there’s a paradox called the Ship of Theseus, that explores the question of identity. The question arises, if we maintain something by replacing its constituent parts, does it remain the same thing? While many hours could be spent debating this, to those whose opinion should matter most, those who use the system, the answer is yes. To them, the identity of the system is bound up in what they do with it, such that it ceases to be the same thing, not when we maintain it but when its function is degraded through neglect.

Common practice, however, separates ownership and interest. Those with the greatest interest in the system typically will not own the budget for work on it. Those owning the budget, will typically be biased towards projects which add value, not maintenance work that represents cost.

Speaking of cost, is 75% of the budget an unreasonable amount for maintenance? How well are the systems meeting the needs of their users? Is quality increasing, decreasing, or holding steady? Was more money spent because of deferred maintenance than would have been spent with earlier intervention? How much business risk is involved? Without this context, it’s extremely difficult to say. It’s understandable that someone outside an organization might lack this information, but even within it, would a centralized IT group have access to it all? Is the context as meaningful at a higher, central level as it is “at the pointy end of the spear”?

Maintaining systems bit by bit, replacing them gradually over time, is likely to be more successful and less expensive, than letting them rot and then having a big-bang re-write. In my opinion, having an effective architecture for the enterprise’s IT systems is dependent on having an effective architecture for the enterprise itself. If the various systems (social and software) are not operating in conjunction, drift and inertia will take care of building your legacy (system).

[Greek Trireme image from Deutsches Museum, Munich, Germany via Wikimedia Commons]

Fixing IT – The Importance of Ownership

My home is your castle?

Organizations that have a centralized budgeting and provision of IT services risk creating a disconnect between those that need technology services and those that provide those services. The first post in this series alluded to the problem – those needing the services have too little control over obtaining those services. If there is only a single provider with finite capacity and a finite budget, then there is the potential for consumers to be left out when demand exceeds supply. Governance processes set up to manage the allocation of services may be ineffective and can lead to dysfunctional behavior when business unit leaders attempt to game the system. Lastly, this situation creates a perverse incentive to gold plate projects in order to compensate for the uncertainty over when the business unit might next get the attention of IT.

The DevOps approach I suggested in my previous post only partially addresses the issues. Having a dedicated team means little if the size, composition, processes, and priorities of that team are outside your control. Much is heard about the concept of “alignment”, but mere alignment is insufficient. As Brenda Michelson noted in “Beware the “Alignment Trap””:

I started to use the term ‘business-IT integration’, because I’m thinking beyond traditional business-IT alignment. Alignment refers to the review and reconciliation of independent activities, in this context the reconciliation of business strategy and plans with IT strategy, architecture and plans.

For business to reap the true value of IT, business and IT must collaborate on the development of strategy, architecture and plans. This collaboration, which should continue through delivery and operations, is business-IT integration.

To achieve the integration Michelson referred to, one should bear in mind Conway’s law: “organizations which design systems … are constrained to produce designs which are copies of the communication structures of these organizations”. The reason to do so is elegantly expressed in Ruth Malan’s paraphrase of it: ” if the architecture of the system and the architecture of the organization are at odds, the architecture of the organization wins”. She goes on to say:

The organizational divides are going to drive the true seams in the system.

The architecture of the system gets cemented in the forms of the teams that develop it. The system decomposition drives team ownership. Then the organizational lines of communication become reflected in the interfaces, with cleaner, better preserved interfaces along the lines where organizational distance increases. In small, co-located teams, short-cuts can be taken to optimize within the team. But each short-cut that introduces a dependency is like rebar in concrete–structurally efficient, but rigid. If the environment changes, demanding new lines to be drawn, the cost becomes clear. The architecture is hard to adapt.

Enterprises are systems, social systems, but systems nonetheless. In the case of business units and IT, you have a social system creating, maintaining, and evolving digital systems. Assuming Conway’s law is true, simpler organizational structures should yield simplified systems (in both the technical and organizational realms). The opposite is also true, poor business architecture can actually reinforce dysfunction and prevent progress. Tom Graves post “Financial-architecture and enterprise-architecture” graphically illustrates just such a case where the accounting structure and approval processes conspired to prevent cost savings:

In fact, the business-case was obvious – or at least, obvious to anyone who’d actually looked at what we’d done. The potential savings in licence-fees alone ran into many millions of dollars a year – let alone the more subtle savings from re-using idle servers or freeing-up unneeded real-estate. By comparison, the costs were tiny: in some cases not much more than a few hours of paperwork, and overall perhaps a few staff-FTE for a few months at most.

But the sting in the tail was that requirement for proof, in terms of the organisation’s standard company-accounts. What we discovered, to our horror, was that the accounts themselves prevented us from doing so: the way those accounts were structured actually made it impossible to make almost any business-case at all for ever shutting anything down.

It should be clear that business structures and processes should be designed to facilitate the execution of business strategy. Part of this is putting technology decision-making into the hands of the business. IT is well positioned to inform those decisions in terms of technical trade-offs, but hardly qualified to decide from a business standpoint. IT Governance: How Top Performers Manage IT Decision Rights for Superior Results by Peter Weill and Jeanne W. Ross (excerpted by David Consulting Group – see table 5) shows that business-driven governance optimizes for profit and growth.

When making decisions about how to use technology in furtherance of business goals is coupled with the responsibility for funding those decisions, customers have greater incentive to make more prudent decisions about their use of technology. Additionally, when those costs are borne by the business unit, both the unit and the enterprise have a clearer picture of its financial state than if those IT costs are accounted for in IT’s budget.

When IT is in the position of advising on and enabling, rather than controlling, use of technology, some benefits accrue. When there’s no need to hide “Shadow IT” from the IT department, business units can take advantage of advice re: security, what other units are doing, etc. The same applies to using outsourcing to augment in-house IT. By being aware of these external capabilities, IT is positioned as a partner to the business unit in attaining its goals, allowing IT to play to its strengths. IT departments that can support business goals as advisers and coordinators of technology in addition to being providers can extend their capabilities with external providers without fear of replacement.