Form Follows Function on SPaMCast 454

SPaMCAST logo

I’m back for another appearance on Tom Cagley’s Software Process and Measurement (SPaMCast) podcast.

This week’s episode, number 454, begins with Tom talking about iteration planning. Jeremy Berriault comes next with a segment on QA team leads and I bat cleanup with a Form Follows Function installment based on my post “Trash or Treasure – What’s Your Legacy?”.

Tom and I discuss the concept of legacy systems: what they are, whether they’re trash or treasure (sometimes both), and how they impact an organization.

You can find all the SPaMCast episodes I’m in under the SPaMCast Appearances category on this blog. Enjoy!

Form Follows Function on SPaMCast 450

SPaMCAST logo

It’s time for another appearance on Tom Cagley’s Software Process and Measurement (SPaMCast) podcast.

This week’s episode, number 450, features Tom’s excellent essay on roadmaps and a Form Follows Function installment based on my post “Holistic Architecture – Keeping the Gears Turning”.

Our conversation in this episode continues with the organizations as system concept, this time from the standpoint of how the social system impacts (often negatively) the software systems the social systems rely on. Specifically, we talk about how an organization that fails to manage itself as a system can lead to an architecture of both the enterprise and its IT that resembles “spare parts flying in formation”. It’s not a good situation, no matter how well made those spare parts are!

You can find all my SPaMCast episodes using under the SPaMCast Appearances category on this blog. Enjoy!

Holistic Architecture – Keeping the Gears Turning

Gears Turning Animation

In last week’s post, “Trash or Treasure – What’s Your Legacy?”, I talked about how to define “legacy systems”. Essentially, as the divergence grows between the needs of social systems and the fitness for purpose of the software systems that enable them, the more likely that those software systems can considered “legacy”. The post attracted a few comments.

I love comments.

It’s nearly impossible to have writers’ block when you’ve got smart people commenting on your work and giving you more to think about. I got just that courtesy of theslowdiyer. The comment captured a critical point:

Agree that ALM is important, and actually also for a different reason – a financial one:

First of all, the cost of operating the system though the full Application Life Cycle (up to and including decommissioning) needs to be incorporated in the investment calculation. Some organisations will invariably get this wrong – by accident or by (poor) design (of processes).

But secondly (and this is where I have seen things go really wrong): If you invest a capability in the form of a new system then once that system is no longer viable to maintain, you probably still need the capability. Which means that if you are adding new capabilities to your system landscape, some form of accruals to sustain the capability ad infinitum will probably be required.

The most important thing is the capability, not the software system.

The capability is an organizational/enterprise concern. It belongs to the social systems that comprise the organization and the over-arching enterprise. This is not to say that software systems are not important – lack of automation or systems that have slipped into the legacy category can certainly impede the enterprise. However, without the enterprise, there is no purpose for the software system. Accordingly, we need to keep our focus centered on the key concern, the capability. So long as the capability is important to enterprise, then all the components, both social and technical, need to be working in harmony. In short, there’s a need for cohesion.

Last fall, Grady Booch tweeted:

Ruth Malan replied with a great illustration of it from her “Design Visualization: Smoke and Mirrors” slide deck:

Obviously, no one would want to fly on a plane in that state (which illustrates the enterprise IT architecture of too many organizations). The more important thing, however, is that even if the plane (the technical architecture of the enterprise) is perfectly cohesive, if the social system maintaining and operating it is similarly fractured, it’s still unsafe. If I thought that pilots, mechanics, and air traffic controllers were all operating at cross purposes (or at least without any thought of common cause), I’d become a fan of travel by train.

Unfortunately, for too many organizations, accidental architecture is the most charitable way to describe the enterprise. Both social and technical systems have been built up on an ad hoc basis and allowed to evolve without reference to any unifying plan. Technical systems tend to be built (and worse, maintained) according to project-oriented mindset (aka “done and run”) leading to an expensive cycle of decay, then fix. The social systems can become self-perpetuating fiefs. The level of cohesion between the two, to the extent that it existed, breaks down even more.

A post from Matt Balantine, “Garbage In” illustrates the cohesion issue across both social and technical systems. Describing an attempt to analyze spending data across a large organization composed of federated subsidiaries:

The theory was that if we could find the classifications that existed across each of the organisations, we could then map them, Rosetta Stone-like, to a standard schema. As we spoke to each of the organisations we started to realise that there may be a problem.

The classification systems that were in use weren’t being managed to achieve integrity of data, but instead to deliver short-term operational needs. In most cases the classification was a drop-down list in the Finance system. It hadn’t been modelled – it just evolved over time, with new codes being added as necessary (and old ones not being removed because of previous use). Moreover, the classifications weren’t consistent. In the same field information would be encapsulated in various ways.

Even in more homogeneous organizations, I would expect to find something similar. It’s extremely common for aspects of one capability to bear on others. What is the primary concern for one business unit may be one of many subsidiary concerns for another (see “Making and Taming Monoliths” for an illustrated example). Because of the disconnected way capabilities (and their supporting systems) are traditionally developed, however, there tends to be a lot of redundant data. This isn’t necessarily a bad thing (e.g. a cache is redundant data maintained for performance purposes). What is a bad thing is when the disconnects cause disagreements and no governance exists to mediate the disputes. Not having an authoritative source is arguably worse than having no data at all since you don’t know what to trust.

Having an idea of what pieces exist, how they fit together, and how they will evolve while remaining aligned is, in my opinion, critical for any system. When it’s a complex socio-technical system, this awareness needs to span the whole enterprise stack (social and technical). Time and effort spent maintaining coherence across the enterprise, rather than detracting from the primary concerns will actually enhance them.

Are you confident that the plane will stay in the air or just hoping that the wing doesn’t fall off?

Trash or Treasure – What’s Your Legacy?

Pirate's burying treasure

The topic of legacy systems is something of a contentious one. In most cases, a legacy is understood to be a good thing. What makes a system “legacy”? Is it a technical or business decision?

A little over a year ago, Greger Wikstrand took a stab at clarifying the term with his post “Legacy systems, a definition”. In the post, he looked at different definitions of what constituted a legacy system, ranging from “any code that is in use” to “outdated technology” to “high technical debt”. The definition he went with, in my opinion, is the most useful:

It should be clear that legacy systems are not about technical considerations. It is about how well the existing system meets and is able to adapt to business needs.

A pair of tweets from Joanna Young that I saw yesterday brought this to mind:

https://twitter.com/jetpack/status/872880714542219266 https://twitter.com/jetpack/status/872881197428477957

Whether or not a system has crossed the line into legacy territory is not a technical decision but a business one. As Greger and Joanna both noted, it’s about fitness for purpose. Technical considerations absolutely have immense bearing on whether the system is able to meet needs. However, they are not the sole determinant.

The standard narrative is for a system to start out “clean” and then rot via neglect and/or ad hoc enhancement. This is certainly a common scenario, but it overlooks the obvious. While failure to maintain a system and its platform will certainly degrade it, keeping the technology components up to date does not ensure that the system will continue to match the needs of those who depend on it. For that matter, it’s easy enough to build a brand new system using the latest and greatest technology that is a legacy right out the gate due to its failure to meet the needs of its stakeholders.

Age of the platform is not a problem; an inability to get support or find people knowledgeable about the platform is a problem. Technical debt in and of itself is not a problem; being impeded or prevented from maintaining/enhancing the system due to technical debt is a problem. This works for any given technical issue – substitute the tangible, stakeholder-oriented result of that technical issue and the point becomes clearer to those with the ability to address them.

The key is not to focus solely on functional aspects nor quality of service and/or technical aspects, but the system as a whole. This requires the participation of the entire set of social systems involved in the creation, maintenance, and usage of the software system. Communication and collaboration across all elements of those social systems is critical to effectively maintaining the software system and the social systems that it enables.

A critically important part of promoting that communication and collaboration is maintaining the cohesion of the social systems involved in creating and maintaining the software system. Where those social systems are ad hoc and episodic, the potential for forming the relationships necessary for effective situational awareness is minimal. IT won’t know about functional gaps until too late and the stakeholders won’t know what their options are for addressing them nor will they have advance notice of impending technical issues.

Social systems create, maintain, and use software systems. Systems that are designed to work together have a better chance of doing so than those that are just thrown together and wished well.

Building a Legacy

Greek Trireme image from Deutsches Museum, Munich, Germany

 

Over the last few weeks, I’ve run across a flurry of articles dealing with the issue of legacy systems used by the U.S. government.

An Associated Press story on the findings from the Government Accountability Office (GAO) issued in May reported that roughly three-fourths of the $80 billion IT budget was used to maintain legacy systems, some more than fifty years old and without an end of life date in sight. An article on CIO.com about the same GAO report detailed seven of the oldest systems. Two were over 56 years old, two 53, one 51, one 35, and one 31. Four of the seven have plans to be replaced, but the two oldest have no replacement yet planned.

Cost was not the only issue, reliability is a problem as well. An article on Timeline.com noted:

Then there’s the fact that, up until 2010, the Secret Service’s computer systems were only operational about 60% of the time, thanks to a highly outdated 1980s mainframe. When Senator Joe Lieberman spoke out on the issue back in 2010, he claimed that, in comparison, “industry and government standards are around 98 percent generally.” It’s alright though, protecting the president and vice president is a job that’s really only important about 60 percent of the time, right?

It would be easy to write this off as just another example of public-sector inefficiency, but you can find these same issues in the private sector as well. Inertia can, and does, affect systems belonging to government agencies and business alike. Even a perfectly designed implemented system (we’ve all got those, right?) is subject to platform rot if ignored. Ironically, our organizations seem designed to do just that by being project-centric.

In philosophy, there’s a paradox called the Ship of Theseus, that explores the question of identity. The question arises, if we maintain something by replacing its constituent parts, does it remain the same thing? While many hours could be spent debating this, to those whose opinion should matter most, those who use the system, the answer is yes. To them, the identity of the system is bound up in what they do with it, such that it ceases to be the same thing, not when we maintain it but when its function is degraded through neglect.

Common practice, however, separates ownership and interest. Those with the greatest interest in the system typically will not own the budget for work on it. Those owning the budget, will typically be biased towards projects which add value, not maintenance work that represents cost.

Speaking of cost, is 75% of the budget an unreasonable amount for maintenance? How well are the systems meeting the needs of their users? Is quality increasing, decreasing, or holding steady? Was more money spent because of deferred maintenance than would have been spent with earlier intervention? How much business risk is involved? Without this context, it’s extremely difficult to say. It’s understandable that someone outside an organization might lack this information, but even within it, would a centralized IT group have access to it all? Is the context as meaningful at a higher, central level as it is “at the pointy end of the spear”?

Maintaining systems bit by bit, replacing them gradually over time, is likely to be more successful and less expensive, than letting them rot and then having a big-bang re-write. In my opinion, having an effective architecture for the enterprise’s IT systems is dependent on having an effective architecture for the enterprise itself. If the various systems (social and software) are not operating in conjunction, drift and inertia will take care of building your legacy (system).

[Greek Trireme image from Deutsches Museum, Munich, Germany via Wikimedia Commons]

Fixing IT – Too Big to Succeed?

Continuing our discussion that I mentioned in my last post, Greger Wikstrand tweeted the following:

I encourage you to watch the video, it’s short (7:39) and makes some important points, which I’ll touch on below.

Serendipity, when it occurs, is a beautiful thing. Serendipity can occur when heads-down order taking is replaced with collaboration. Awareness of business concerns on the part of IT staff enhances their ability to work together with end users to create effective solutions. Awareness, however, is insufficient.

Woody Zuill‘s early remarks about the bureaucracy that gets in the way of that serendipity struck a nerve. Awareness of how to address a need does little good if the nature of the organization either prevents the need from being addressed. Equally bad is when a modest need must be bloated beyond recognition in order to become “significant” enough for IT intervention. Neither situation works well for the end users, IT, or the business as a whole.

So what’s needed to remedy this?

In “Project lead time”, John Cook asserted that project lead team was related to company size:

A plausible guess is that project lead time would be proportional to the logarithm of the company size. If a company with n employees has a hierarchy with every manager having m subordinates, the number of management layers would be around logm(n). If every project has to be approved by every layer of management, lead time should be logarithmic in the company size. This implies huge companies only take a little longer to start projects than medium-sized companies, and that doesn’t match my experience.

While I agree in principle, I suspect that the dominant factor is not raw size, but the number of management layers Cook refers to. In my experience, equally sized organizations can be more or less nimble depending on the number of approvals and the reporting structure of the approvers. Centralized, shared resources tend to move slower than those that are federated and dedicated. Ownership increases both customer satisfaction and their sense of responsibility for outcomes. Thus, embedding IT staff in business units as integral members of the team seems a better choice than attempting to “align” a shifting set of workers temporarily assigned to a particular project.

The nature of project work, when combined with a traditional shared resource organization, stands as a stumbling block to effective, customer-centric IT. IT’s customers (whether internal, external, or both) are interested in products, not projects. For them, the product isn’t “done” until they’re no longer using it. While I won’t go so far as to say #NoProjects, I share many of the same opinions. Using projects and project management to evolve a product is one thing, I’ve found project-centric IT to be detrimental to all involved.

Having previously led embedded teams (which worked as pseudo-contractors: employed by IT, with the goals and gold coming from the business unit), I can provide at least anecdotal support to the idea that bringing IT and its customers closer pays dividends. Where IT plays general contractor, providing some services and sub-contracting other under the direction of those paying the bills, IT can put its expertise to use meeting needs by adding value rather than just acting as order-takers. This not only allows for the small wins that Woody Zuill mentioned, but also for integrating those small wins into the overall operation of the business units, reducing the number of disconnected data islands.

Obviously there are some constraints around this strategy. Business units need to be able to support their IT costs out of their own budget (which sounds more like a feature than a bug). Likewise, some aspects of IT are more global than local. The answer there, is matching the scope of the IT provider unit with the scope of the customer unit’s responsibility. Federating and embedding allows decisions to be made closer to the point of impact by those with the best visibility into the costs and benefits. Centralization forces everything upwards and away from those points.

Operating as a monolithic parallel organization that needs to be “aligned” to the rest of the organization sounds like a system that’s neither effective nor efficient, certainly not both. When the subject is the distance between the decision and its outcome, bigger isn’t better.