“Dealing with “Technical Debt”” on Iasa Global

The term “technical debt” has a tendency to evoke an emotional response. Some people react puritanically – “technical debt” means sloppy code; sloppy code is sin; failing to call sin “sin” is condoning sin; to the stake with the heretic! Others will contend that technical debt solely refers solely to conscious trade-offs and by definition excludes code that is poorly written or designed.

Read “Dealing with “Technical Debt”” on Iasa Global for my take on how broadly “technical debt” should be defined.

Advertisement

#4U2U – Canned Competency, Values & Pragmatism

Not quite two years ago, I put up a quick post entitled “The Iron Law of Tools”, which in its essence was: “that which does for you, can do to you” (whence comes the #4U2U in the title of this post). That particular post focused on ORMs (Entity Framework to be specific), but the warning equally applies to libraries and frameworks for other technical issues as well as process, methodology and techniques.

Libraries, frameworks, and processes (“tools” from this point forward) can make things easier by allowing you to concentrate on what to do rather than how to do it (via high-level abstractions and/or conventions). However, tools are not a substitute for understanding. Neither the Law of Unintended Consequences nor Murphy’s Law have been repealed. Without an adequate understanding of how something works, you cannot assess the costs of the trade-offs that are being made (and there are trade-offs involved, you can rely on that). Understanding is likewise necessary to recognize and fix those situations where the tool causes more problems than it solves. As Pawel Brodzinski observed in his post “A Fool With a Tool Is Still a Fool”:

Any time a discussion goes toward tools, any tools really, it’s a good idea to challenge the understanding of a tool itself and principles behind its successes. Without that shared success stories bear little value in other contexts, thus end result of applying the same tools will frequently result in yet another case of a cargo cult. While it may be good for training and consulting businesses (aka prophets) it won’t help to improve our organizations.

A fool with a tool will remain a fool, only more dangerous since now they’re armed.

Pawel’s point regarding cargo cults is particularly important. Lack of understanding of how a particular effect proceeds from a given cause often manifests as dogmatic assertions in defense of some “universal truth”. The closest thing I’ve found to a universal truth of software development is that it’s very unlikely that anything is universally applicable to every context.

It’s dangerous to conflate adherence to a tool with one’s core values, such that anyone who disagrees is “wrong” or “deluded” or “unprofessional”. That being said, values can provide a frame of reference in understanding someone’s position in regard to a tool. In “The TDD Divide: Everyone is Right”, Cory House addresses the current dispute over Test-Driven Development and notes (rightly, in my opinion):

The world is a messy place. Deadlines loom, team skills vary widely, and the impact of bugs varies greatly by industry. Ultimately, we write software to make money and solve problems. Tests are a tool that help us do both. Consider the context to determine which testing style fits for your project.

Uncle Bob is right. Quality matters. Separation of concerns and unit testing help assure the utmost quality, speed, and flexibility.

DHH is right. Sometimes the cost of unit tests exceed their benefit. Some of the benefit of automated testing can be achieved through automated integration testing instead.

You need to understand what a tool offers and what it costs and the result of that equation in relation to what’s important in your context. With that understanding, you can make a rational choice.

Why does software development have to be so hard?

A series of 8 tweets by Dan Creswell paints a familiar, if depressing, picture of the state of software development:

(1) Developers growing up with modern machinery have no sense of constrained resource.

(2) Thus these developers have not developed the mental tools for coping with problems that require a level of computational efficiency.

(3) In fact they have no sensitivity to the need for efficiency in various situations. E.g. network services, mobile, variable rates of change.

(4) Which in turn means they are prone to delivering systems inadequate for those situations.

(5) In a world that is increasingly networked & demanding of efficiency at scale, we would expect to see substantial polarisation.

(6) The small number of successful products and services built by a few and many poor attempts by the masses.

(7) Expect commodity dev teams to repeatedly fail to meet these challenges and many wasted dollars.

(8) Expect smart startups to limit themselves to hiring a few good techies that will out-deliver the big orgs and define the future.

The Fallacies of Distributed Computing are more than twenty years old, but Arnon Rotem-Gal-Oz’s observations (five years after he first made them) still apply:

With almost 15 years since the fallacies were drafted and more than 40 years since we started building distributed systems – the characteristics and underlying problems of distributed systems remain pretty much the same. What is more alarming is that architects, designers and developers are still tempted to wave some of these problems off thinking technology solves everything.

Why?

Is it really this hard to get it right?

More importantly, how do we change this?

In order to determine a solution, we first have to understand the nature of the problem. Dan’s tweets point to the machines developers are used to, although in fairness, those of us who lived through the bad old days of personal computing can attest that developers were getting it wrong back then. In “Most software developers are not architects”, Simon Brown points out that too many teams are ignorant of or downright hostile to the need for architectural design. Uncle Bob Martin in “Where is the Foreman?”, suggests the lack of a gatekeeper to enforce standards and quality is why “our floors squeak”. Are we over-emphasizing education and underestimating training? Has the increasing complexity and the amount of abstraction used to manage it left us with too generalized a knowledge base relative to our needs?

Like any wicked problem, I suspect that the answer to “why?” lies not in any one aspect but in the combination. Likewise, no one aspect is likely, in my opinion, to hold the answer in any given case, much less all cases.

People can be spoiled by the latest and greatest equipment as well as the optimal performance that comes for working and testing on the local network. However, reproducing real-world conditions is a bit more complicated than giving someone an older machine. You can simulate load and traffic on your site, but understand and accounting for competing traffic on the local network and the internet is a bit more difficult. We cannot say “application x will handle y number of users”, only that it will handle that number of users under the exact same conditions and environment as we have simulated – a subtle, but critical difference.

Obviously, I’m partial to Simon Brown’s viewpoint. The idea of a coherent, performant design just “emerging” from doing the simplest thing that could possibly work is ludicrous. The analogy would be walking into an auto parts store, buying components individually, and expecting them to “just work” – you have to have some sort of idea of the end product in mind. On the other hand, attempting to specify too much up front is as bad as too little – the knowledge needed is not there and even if it were, a single designer doesn’t scale when dealing with any system that has a team of any real size.

Uncle Bob’s idea of a “foreman” could work under some circumstances. Like Big Design Up Front, however, it doesn’t scale. Collaboration is as important to the team leader as it is to the architect. The consequences of an all-knowing, all-powerful personality can be just as dire in this role as for an architect.

In “Hordes of Novices”, Bob Martin observed “When it’s possible to get a degree in computer science without writing any code, the quality of the graduates is questionable at best”. The problem here is that universities are geared to educate, not train. Just because training is more useful to an employer (at least in the short term), does not make education unimportant. Training deals with this tool at this time while how to determine which tool is right for a given situation is more in the province of education. It’s the difference between how to do versus how to figure out how to do. Both are necessary.

As I’ve already noted, it’s a thorny issue. Rather than offering an answer, I’d rather offer the opportunity for others to add to the conversation in the comments below. How did we get here and how do we go forward?

What do we mean by “architecture”?

Every systematic development of any subject ought to begin with a definition, so that everyone may understand what the discussion is about.

Marcus Tullius Cicero (196BC ‒ 16BC), De Officiis, Book 1, Moral Goodness (h/t to Glen Alleman, October 8, 2013 Quote of the Day)

One of the joys of the English language is the overloading of words with multiple meanings. While it’s not hard to start an argument over technical issues under normal circumstances, having multiple definitions to conflate makes it that much easier. Take, for example, the word “architecture”. While different dictionaries give slightly different definitions (e.g. Merriam-Webster Dictionary.com and Wiktionary), the common set of definitions boil down to:

• the practice of architecture (i.e. the discipline of architectural design)
• an architectural style (e.g. layered, RESTful, etc.)
• the result of architectural work (i.e. the architecture of a thing – the “decisions of consequence” that define the structure and behavior of a system as well as constraining its future evolution)

While there is a relationship between these concepts, a person practicing architecture might create an architectural design using one or more architectural styles, there is enough differentiation to cause trouble if we don’t clarify our terms. Far from being an academic exercise, paying attention to semantics can actually save time and frustration when there is ambiguity present.

Architecture versus Design

There was interesting debate on what is an architect, the difference between design and architecture (no one really knows…
(Ilan Kirschenbaum, discussing a meeting of the Software Craftsmanship in Israel group)

All architecture is design but not all design is architecture. Architecture represents the significant design decisions that shape a system, where significant is measured by cost of change.
(Grady Booch, as quoted by nearly everyone)

In the inaugural post of Form Follows Function, I noted the definition of architecture that I felt best applied across the realms of Application, Solution, and Enterprise Architecture. While researching the post “What’s driving your architecture?”, I found considerable discussion of the difference between architecture and design (in the sense of design decisions that are architecturally significant versus those that are not). Although the distinction might seem to be academic at first glance, it cuts to the heart of questions around the value of architecture and architects, particularly under Agile processes. This makes the confusion alluded to above by Ilan Kirschenbaum doubly worrisome.

Simon Brown, building on Booch’s comment above, provides important insights on the concept of design in “Disambiguating architecture”. Design “…exists to resolve a force on the system”, where force represents “…cost, scope, resources, timescales, requirements – anything you wish to take into consideration as part of your design selection”. He also notes that “…a design is one of a number of possible choices for resolving a particular force on the system”. Thus design, architectural or not, can be defined as making the optimal choice to satisfy the various forces impacting the matter at hand.

Rick Kazmon and Amnon Eden, writing for the Carnegie Mellon Software Engineering Institute, identified locality as the defining characteristic of architectural design in their article “Defining the Terms Architecture, Design, and Implementation”. Non-local specifications, applying to the system as a whole, constitute architecture in their opinion. According to Kazmon and Eden, “More generally, for each specification we should be able to determine whether it is a design statement, describing a purely local phenomenon (and hence of secondary interest in architectural documentation, discussion, or analysis), or whether it is an instance of an underlying, more general rule”.

Eden and Kazman, joined by Yoram Hirshfeld, further expanded on this definition in “Abstraction Classes In Software Design”, published in IEE Software Vol. 153, No. 4 (Aug. 2006). The authors identify global design decisions as strategic and local design decisions as tactical:

Strategic design statements [11] articulate design decisions that determine the primary behavioural and structural properties of a program (software system). Strategic decisions address global, system-wide concerns and carry the most consequential implications. Strategic decisions include the choice of programming paradigm [36] (‘object-oriented programming’), architectural style [17] (‘pipes and filters’), application framework [24] (‘Microsoft Foundation Classes’), component-based software engineering standards [40] (‘Enterprise JavaBeans’), and design principles (‘universal base class’), as well as assumptions that may lead to architectural mismatch [18] (‘The Softbench Broadcast Message Server expected all of the components to have a graphical user interface’) and law governed regularities [28] (‘every class in the system inherits from class C’). Because of the consequences they carry, Strategic decisions must be made early in the software development process and should be established explicitly before any detailed design is carried out.

In contrast, Tactical design [11] statements articulate design decisions that are concerned with a specific module. Tactical decisions often describe a pattern of correlations between one collection of modules (objects, procedures, classes etc.) and another. Intuitively speaking, Tactical statements are ‘local’ in the sense that their scope is limited to some part of the program and not outside it. Tactical decisions include the choice of design patterns [16] (‘Factory Method’), refactorings [15] (‘Replace Conditional With Polymorphism’), and programming idioms [5] (‘counted pointer’), and usually are taken much later than strategic design decisions in the software development process.

Although I broadly agree with the general rule of global = strategic and local = tactical, it should not be considered an absolute. It is possible for one segment of a system to have aspects that strongly influence the overall architecture (justifying Booch’s criteria for significance). This is the reason that I do not hold the opinion that functional requirements are design and non-functional are architecture.

Arnon Rotem-Gal-Oz, architecture manager at Nice Systems, identifies several aspects of architecture in his presentation “Software Architecture”. Architecture comprises the major components of a system; their relationships and interactions. Data and behavior of these major components is only relevant to architecture from the point of view of the interactions between these components. Like Booch, Rotem-Gal-Oz characterizes the architecture as being the most difficult parts of the system to change. Like Eden, Kazman, and Hirshfeld, he defines architecture as the design decisions that must be made earliest in the process.

The value of strategic, foundational design decisions should be obvious. Since these aspects of the system are both difficult and costly to change, there should be considerable incentive to make optimal choices. As noted by Kevlin Henney in “What is Software Architecture?” on developerFusion:

The issue is not whether or not a system has architecture, but whether its architecture is good or not. All systems have architecture, whether intentional or accidental, explicit or implicit, good or bad – even “no architecture” systems have architecture!

Having related architecture to cost of change, we have a simple way of evaluating architectural quality. It is not that a good architecture makes a system cheap to create; it is that it is also cheap to evolve. In other words, a good architecture is one in which the significance of design decisions is minimised. You are free to change many aspects of a system without the fear that such changes are challenging, costly or create a cascade of bugs. A good architecture is therefore a sustainable one.

Given the criticality of these design decisions, it makes little sense to leave them to chance. Sustainable, flexible architecture is unlikely to emerge organically. Sufficient planning and coordination will be needed from someone in the architect role (regardless of whether the role is full or part-time). Localized design decisions can then be made within that framework in a decentralized manner.

Occasionally I hear people still raising the agile mantra against Big Design/Requirements Up Front. The thing is that Agile Manifesto never said to intentionally bury your head in the sand with regards to the purpose of the system. It was a push-back against spending months in analysis without anything but documents coming out, but the goal was to reach a middle ground. Nobody ever said “no design up-front” or “no requirements up front”.
(Udi Dahan from “A CQRS Journey – with and without Microsoft”)

So what exactly does an Architect do?

It’s not unusual in the IT world to have difficulty explaining what you do to your parents. It is, however, a cause for concern when your peers, even your superiors, don’t have a clear understanding. Unfortunately, that’s the case with architects right now.

Part of the problem is terminology. Terms like Enterprise Architect, Solutions Architect, Technical Architect, Information Architect, Business Architect, Database Architect, Software Architect, Network Architect, Systems Architect, User Experience Architect, Security Architect, Application Architect, and Infrastructure Architect are thrown around without authoritative definition and little agreement as to what they mean. I’ve yet to see a comprehensive taxonomy and I won’t be attempting one here. Instead, I hope to highlight some recent work in the area and contribute something of my own to help increase understanding.

The best place to start, is with some definitions. As I noted in the blog’s inaugural post, Wikipedia defines “architecture” as “…both the process and product of planning, designing and construction”. Merriam-Webster’s second definition for “architect” is “a person who designs and guides a plan or undertaking”. These definitions, in my opinion, apply regardless of the type of architect being described. All architects plan, design, and guide.

I’ve found three articles that, in my opinion, do a good job of illustrating how different architect roles relate to each other and what those roles do: IASA Sweden’s Study of Architect Roles, ArchiTech Consulting LLC’s “Enterprise Architecture .vs. Collection of Architectures in Enterprise”, and Igor Lobanov’s “Differences Between Architecture Roles”. I conclude with my thoughts on the subject.

IASA Sweden’s Study of Architect Roles

The Sweden chapter of Iasa (originally the International Association of Software Architects) commissioned a study of Architect Roles that was published in the April 2008 edition of The Architecture Journal. The article dealt with four roles: Enterprise Architect, Business Architect, Solution Architect, and Software Architect. The roles are described as overlapping (see image below) and differing in depth and breadth of knowledge as well as scope of interest. Unfortunately, the article ignores two architectural domains: Technology Architecture and Information Architecture. Additionally, the description of the Enterprise Architect is too IT-centric.

ArchiTech Consulting LLC’s “Enterprise Architecture .vs. Collection of Architectures in Enterprise”

“Enterprise Architecture .vs. Collection of Architectures in Enterprise” by Yan Zhao, PhD, was posted on the ArchiTech Consulting LLC blog on November 21. While the article was mainly focused on the Zachman Framework and how it defines a collection of architectures within an enterprise, it also illustrated the relationship of architectures (and by extension, architects) within that framework (see image below). While it leaves out the overlap, it does a much better job of showing the level at which each role works and it includes the Information Architecture domain.

Igor Lobanov’s “Differences Between Architecture Roles”

Igor Lobanov posted “Differences Between Architecture Roles” on his blog on September 19. While it doesn’t touch on the Business or Information Architecture domains, it provides the best visual representation (see image below) of the depth, breadth, and scope of Enterprise, Solution, and Technical Architects. He also captures the overlap inherent in the roles. In his model, the Technical Architect (correctly, in my opinion) encompasses both the Application Architecture and Technology Architecture domains. Additionally, Igor’s descriptions of the three roles are excellent, detailing the concerns that each role deals with.

My Thoughts

The fact that we are dealing with roles rather than positions complicates matters, because one person can undertake many roles. This is why the overlap aspect is one I consider important. I would hazard a guess that most Solution Architects serve double duty as Technical Architects (generally Application Architects) in small and medium-sized organizations. For that matter, in many cases the Solutions Architect/Application Architect may also double as a senior developer/technical lead (which has both positive and negative effects, but that’s a post for another day). As shown by all three articles, however, the roles share common characteristics differing mainly by depth and breadth of knowledge as well as scope of interest.

For the most part, Information Architecture and Business Architecture receive short shrift in the articles linked to. This is unfortunate as these domains are becoming increasingly important (Information Architecture in particular). I suspect, given recent trends, that those roles will become more common and better defined in the very near future.

Security Architecture is another neglected, but critical area. There’s still some debate as to whether it is another domain or a cross-cutting concern. That’s a debate I won’t be able to settle here.

My concept is a blend of the three articles I’ve referenced. From a high level, I like the overlapping circles of the Iasa model, but I see the Enterprise Architect as encompassing all four domains (Business, Information, Application, and Technology), with the Solutions Architect occupying a similar role on a smaller scale (fewer domains and less organizational scope). The remaining roles I listed, in my opinion, shake out as follows:

• Technical Architect – this is more a category than a role.  Both Application Architects and Technology Architects would be considered Technical Architects
• Software Architect – a synonym for Application Architect.
• Infrastructure Architect – a synonym for Technology Architect.
• Network Architect – a specialization of the Infrastructure Architect role.
• Systems Architect – Wikipedia’s definition is synonymous with Solutions Architect, but the usage I’ve seen more commonly is that of an Infrastructure Architect dealing with servers and operating systems.
• User Experience Architect – a specialization of the Application Architect dealing with user interface design and usability.
• Database Architect – I view this role as a specialization of the Application Architect (big surprise: an Application Architect views the database as a component of the application), but I see it commonly falling into the Infrastructure space.

There are, of course, roles that I’ve left out. SOA Architects, Storage Architects and Active Directory/LDAP Architects come to mind, but I’ll leave it as an exercise for the reader to fit them into the framework above. It’s my hope that the framework itself is useful for understanding what to expect from these roles.

Like it or not, you have an architecture (in fact, you may have several)

What better way to inaugurate the blog than by defining what is meant by architecture and why you should care?

Wikipedia’s definition: “Architecture is both the process and product of planning, designing and construction” is both concise and comprehensive (which is commendable from an architectural standpoint). It applies to all the various types of architecture, both physical and virtual. Regardless of the size of your organization, you will have multiple “products of planning, designing and construction”.

If you have a laptop, a cable modem and a wireless router, then you have a network architecture. Add in Office, QuickBooks and Gmail and you now have a solutions architecture (with a cloud component, no less). Of course, the data from those applications provides you with an information architecture. Combining those elements with the structure of your business: defining your current state, assessing how well you the technology/data/applications align with the business, mapping out where you want to be etc. and you have an enterprise architecture.

Does this mean that all enterprises, regardless of size, need formal architecture programs and, of course, the architects to run them? Obviously, the answer is “no”. There are far too many successful businesses without such programs for that to be the case. Even if the answer were “yes”, those too small to afford the expense would have to choose between making do and giving up (and I doubt the latter is really an option).

The important thing to take away is that “architect” is the name of a role, not necessarily a position. Not everyone “doing architecture” is a full time architect. Many small business owners perform the functions without thinking of themselves as enterprise architects. What is necessary is that some care is directed toward architectural concerns, because there is a critical omission in the definition above: Architecture should be both the process and product of planning, designing and construction, although it may just be the product of construction.

With one notable exception, most do not intentionally create architectural nightmares. They just evolve that way. You start simple, with an add-on here and an enhancement there; suddenly what was once simple is now rambling and incoherent. In such a situation, maintaining normal function may well consume more and more precious time and attention. This is where an up front focus on architecture pays dividends. Managed growth and change is far easier to deal with (not to mention cheaper) than the organic variety.