Square Pegs, Round Holes, and Silver Bullets

Werewolf

People like easy answers.

Why spend time analyzing and evaluating when you can just take some thing or some technique that someone else has already put to use and be done with it?

Why indeed?

I mean, “me too” is a valid strategy, right?

And we don’t want people to get off message, right?

https://twitter.com/jetpack/status/844195111407943680

And we can always find a low cost, minimal disruption way of dealing with issues, right?

https://twitter.com/jetpack/status/845323751545884675

I mean, after all, we’ve got data and algorithms, and stuff:

The thing is, actions need to make sense in context. Striking a match is probably a good idea in the dark, but it’s probably less so in daylight. In the presence of gasoline fumes, it’s a bad idea regardless of ambient light.

A recent post on Medium, “Design Sprints Are Snake Oil” is a good example. Erika Hall’s title was a bit click-baitish, but as she responded to one commenter:

The point is that the original snake oil was legitimate and effective. It ended up with a bad reputation from copycats who over-promised results under the same name while missing the essential ingredients.

Sprints are legitimate and effective. And now there is a lot of follow-up hype treating them as a panacea and a replacement for other types of work.

Good things (techniques, technologies, strategies, etc.) are “good”, not because they are innately right, but because they fit the context of the situation at hand. Those that don’t fit, cease being “good” for that very reason. Form absent function is just a facade. Whether it’s business strategy, management technique, innovation efforts, or process, there is no recipe. The hard work to match the action with the context has to be done.

Imitation might be the sincerest form of flattery, but it’s a really poor substitute for strategy.

Strategic Tunnel Vision

Mouth of a Tunnel

 

Change and innovation are topics that have been prominent on this blog over the last year. In fact, Greger Wikstrand and I have traded a total of twenty-six posts (twenty-seven counting this one) on the subject.

Greger’s last post, “Successful digitization requires focus on the entire customer experience – not just a neat app” (it’s in Swedish, but it translates well to English), discussed the critical nature of customer experience to digital innovation. According to Greger, without taking customer experience into account:

One can make the world’s best app without getting more, more satisfied and profitable customers. It’s like trying to make a boring games more exciting by spraying gold paint on the playing pieces.

Change and innovation are not the same thing. Change is inevitable, innovation is not (with a h/t to Tom Cagley for that quote). As Greger pointed out in his latest article, to get improved customer experience, you need depth. Sprinkling digital fairy dust over something is not likely result in innovation. New and different can be really great, but new and different solely for the sake of new and different doesn’t win the prize. Context is critical.

If you’ve read more than a couple of my posts, you’ve probably realized that among my rather varied interests, history is a major one. I lean heavily on military history in particular when discussing innovation. This post won’t break with that tradition.

The blog Defense in Depth, operated by the Defence Studies Department, King’s College London, has published two posts this week dealing with the Suez Crisis of 1956, primarily in terms of the Anglo-French forces. One deals with the land operations and the other with naval operations. They struck a chord because they both illustrated how an overreaction to change can have drastic consequences from the strategic level down to the tactical.

Buying into a fad can be extremely expensive.

The advent of the nuclear age at the end of World War II dramatically transformed military and political thought. The atomic bomb was the ultimate game-changer in that respect. In the time-honored tradition, the response was over-reaction. “Atomic” was the “digital” of the late 40s into the 60s. They even developed a recoilless gun that could launch a 50 pound nuclear warhead 1.25-2.5 miles. “Move fast and break things” was serious business back in the day.

This extreme focus on what had changed, however, led to a rather common problem, tunnel vision. Nuclear capability became such an overarching consideration that other capabilities were neglected. Due to this neglect of more conventional capabilities, the UK’s forces were seriously hampered in their ability to perform their mission effectively. Misguided thinking at the strategic level affected operations all the way down to the lowest tactical formations.

It’s easy to imagine present-day IT scenarios that fall prey to the same issues. A cloud or digital initiative given top priority without regard to maintaining necessary capabilities could easily wind up failing in a costly manner and impairing the existing capability. It’s important to understand that time, money, and attention are finite resources. Adding capability requires increasing the resources available for it, either through adding new resources or freeing up existing ones by reducing the commitment to less important capabilities. If there is no real appreciation of what capabilities exist and what the relative value of each is, making this decision becomes a shot in the dark.

Situational awareness across all levels is required. To be effective, that awareness must integrate changes to the context while not losing sight of what already was. Otherwise, to use a metaphor from my high school football days, you risk acting like a “blind dog in a meat-packing plant”.

Storm Clouds: DropBox’s Back to the Future Moment

Rembrant's Christ in the Storm on the Lake of Galilee

One of the big news items from last week was DropBox’s announcement that it had brought its file storage infrastructure in-house, moving (mostly) away from AWS:

Years ago, we called Dropbox a “Magic Pocket” because it was designed to keep all your files in one convenient place. Dropbox has evolved from that simple beginning to become one of the most powerful and ubiquitous collaboration platforms in the world. And when our scale required building our own dedicated storage infrastructure, we named the project “Magic Pocket.” Two and a half years later, we’re excited to announce that we’re now storing and serving over 90% of our users’ data on our custom-built infrastructure.

Given both the massive scope of the endeavor and the massive repudiation of what’s becoming more and more common infrastructure practice, Wired.com‘s article is appropriately titled: “The Epic Story of Dropbox’s Exodus From the Amazon Cloud Empire”. According to that article:

In essence, they built their own Amazon S3—except they tailored their software to their own particular technical problems. “We haven’t built a like-for-like replacement,” (Dropbox engineering VP Aditya) Agarwal says. “We’ve built something that is customized for us.”

Did DropBox make the right decision?

Only time will tell. In truth, it will probably be easier to tell if it was the wrong decision than if it was the right one. Poor choices tend to be more absolute, good choices less clear-cut.

DropBox has also spent more than two years developing and proving their infrastructure accord to their announcement. Given that file storage is DropBox’s core business and recognizing the scale they operate at (500 million users and 500 petabytes of data according to an article on CIO.com), the idea that they should control their own infrastructure makes sense. This is particularly true given Agarwal’s statement that “We’ve built something that is customized for us.” While I can’t say whether the decision is right, I can say that it is a reasonable one.

That’s not the same, however, as saying that everyone should operate the own infrastructure. Context matters. Simon Wardley‘s tweet sums it up nicely (where “nicely” is defined as “snarkily”):

Emulating DropBox when you’re not in the storage business, when you don’t have the volume (nor, most likely, the budget), and when you haven’t done the homework to prove it out, makes no sense. It would be like rowing a dinghy out into a stormy ocean because the oil tanker that just left port is doing fine.

Context matters.

Don’t Teach Coding

Before the pitchforks and torches come out, allow me to clarify – don’t just teach coding.

Jeff Susna and Mark M captured the essence of what I’m talking about:

https://twitter.com/jetpack/status/638453985880338432

The article Mark M referenced, “Is coding the new literacy?”, had this to say:

So you might be forgiven for thinking that learning code is a short, breezy ride to a lush startup job with a foosball table and free kombucha, especially given all the hype about billion-dollar companies launched by self-taught wunderkinds (with nary a mention of the private tutors and coding camps that helped some of them get there). The truth is, code—if what we’re talking about is the chops you’d need to qualify for a programmer job—is hard, and lots of people would find those jobs tedious and boring.

But let’s back up a step: What if learning to code weren’t actually the most important thing? It turns out that rather than increasing the number of kids who can crank out thousands of lines of JavaScript, we first need to boost the number who understand what code can do. As the cities that have hosted Code for America teams will tell you, the greatest contribution the young programmers bring isn’t the software they write. It’s the way they think. It’s a principle called “computational thinking,” and knowing all of the Java syntax in the world won’t help if you can’t think of good ways to apply it.

Whether you call it computational thinking or computer literacy, understanding the high-level basics of the technology is what is useful. Conflating the ability to create Hello World in JavaScript with that understanding is both simplistic and counter-productive. Chris Granger, in “Coding is not the new literacy”, observed:

Being literate isn’t simply a matter of being able to put words on the page, it’s solidifying our thoughts such that they can be written. Interpreting and applying someone else’s thoughts is the equivalent for reading. We call these composition and comprehension. And they are what literacy really is.

Languages come and go. Technologies evolve. Concentrating exclusively on specific languages and technologies risks creating future obsolescence. The underlying concepts that must be understood to effectively use them, however, have longer lives. It’s the 21st century equivalent of the difference between giving someone a fish and teaching them to fish. Erik Dietrich in “Don’t Learn to Code — Learn to Automate” points out the broader utility of computer literacy:

It’s obtuse to suppose that a prerequisite for every job in the future will be the ability to implement sophisticated, specialized computer applications. But it’s not at all obtuse to suppose that, given the ubiquity of computing, a prerequisite for every job in the future will be the ability to recognize which tasks are better suited for humans and which for computers. Learn at least to recognize which parts of your job are a poor use of your time. After that, perhaps learn to use your ingenuity and creativity to automate using the tools that you know (such as googling for solutions, leveraging apps, etc). And, if you’ve come that far, maybe it’s time to roll up your sleeves and take the plunge into learning to code a little bit to help you along.

The need for greater numbers of people with computer literacy is real, as is the need for greater diversity within the ranks of those who work with technology. It serves no one to misrepresent the skills needed or the nature of those skills. People who know how to google for code but lack the ability to understand and evaluate what they get back are no better off than if they had never seen a computer. We owe them, and ourselves, better.

Cause and Effect – Cargo Cults and Carts Before Horses

Sometimes our love of shortcuts can make us really stupid. Take, for example, the idea of “Fail Fast”. As Jeff Sussna observed in his post “Rethinking Failure”, “Suddenly failure is all the rage.” He also noted:

By itself, failure is anything but good. Making the same mistake over and over again doesn’t help anyone. Failure leads to success when I learn from it by changing my behavior or understanding in response to it. Even then, it’s impossible to guarantee that my response will in fact lead to success. The validity of any given response can only be evaluated in hindsight.

Dan McClure, in “Why the “Fail Fast” Philosophy Doesn’t Work”, agreed:

If your only strategy for exploring the unknown is to pick up rocks and look underneath, then the more rocks you turn over the better. The problem is that for real world innovations, test and reject doesn’t scale well. For disruptive ideas with the potential to make a difference in the market there are lots and lots of rocks.

The value in “Fail Fast” lies in the “Fast” part; there’s no magic in the “Fail”. If you’re going to fail, finding out about it sooner, rather than later, is less costly. Less costly, however, is a far cry from best. Succeeding obviously works much better than failing fast, meaning that methods which allow you to evaluate without incurring the time, pain, and expense of a failure are a better choice when available.

Another example of this phenomenon is what Matt Balantine recently referred to as “investor-centric” development:

https://twitter.com/jetpack/status/623200517607583744

That, of course, creates an interesting rabbit hole – investors chasing products that will be “hot” and products designed to appeal to investors rather than customers (which would result in the product becoming “hot”). Matt’s comment from his post “What if the answer isn’t software?” applies, “I’ve no doubt that we are seeing real issues and opportunities being ignored in the pursuit of the rainbow-pooping unicorns.”

Yet another example of magical thinking is via believing in the “Great Man Theory”. People like Steve Jobs and Elon Musk have achieved great things, but as a result of what they did, not who they are. Divorced from their context, it’s a hard sell to argue that they would be equally successful.

Effectiveness is more likely to come from systems thinking than magical thinking. Understanding cause and effect as well as interrelationships and context makes the difference between rational decision-making and superstition.

Cargo Cult Architecture

According to Mark Little, Red Hat VP of Engineering, the microservice backlash has arrived, coming from “people who were really pushing it at the beginning and who are now just starting to realise it’s not all sunshine and roses, or people who never felt the need for it at all”. The Twitterverse seems to agree:

This post, however, is less about microservices, and more about what their rise and fall (and, no doubt, recovery as we violently discover equilibrium) says about software development as a discipline.

As Sander Mak observed in his post “On monoliths, microservices and critical thinking” (h/t Paul Bakker):

What does it mean if public software engineering opinion flips 180 degrees in a matter of weeks? It’s too easy to chalk it all up to people needing authority figures. Yes, I know: not everybody was all over microservices. But you have to admit there’s something fundamentally unsound going on here.

This is hardly a new problem. The same Mark Little mentioned in the opening wrote an article for InfoQ almost three years ago titled “IT Values Technologies Over Thought” where he stated “If the people delivering the implementations that are supposed to be solutions to business problems aren’t looking beyond the hype and considering alternatives, especially when those alternatives may have been tried and tested for many years, then we are in for some very interesting times ahead”.

It’s a known problem. We even laugh at articles that trade on our tendency to jump from silver bullet to silver bullet (although I’m not sure if that laughter is based on sangfroid or fatalism).

It’s not even a problem that’s exclusively ours. An article in Forbes, “Why So Many Management Strategies Become Fads That Fade Away”, refers to it as “idea surfing”. When complexity, unrealistic expectations, cultural resistance, or poor fit lead to management souring on the current strategy du jour, there’s always a shinier object just down the road that promises to be the recipe for success.

Accord to “Rats Can Be Smarter Than People” in January’s Harvard Business Review, our predilection for easy answers is deeply rooted (emphasis added):

Our rule-based system was an evolutionary development: How do you tell if a berry is good for eating? You learn that this small red one is good, and then you save energy by bypassing the ones of a different shape or color. So our brains have been conditioned to look for rules. We’re taught them in school, at work, and by our parents, and we can make many good decisions by applying the ones we’ve learned. But in other situations there’s too much going on for simple rules to work, and that’s when information integration learning has to kick in. Think of a radiologist evaluating an X-ray. If you ask him what rules he uses to determine whether a spot is cancer, he’d probably have a hard time verbalizing them. He’s learned from labeled examples in medical school and his own experience, and then developed an instinct for identifying cancerous spots based on what he’s seen before. Another example that comes to mind is a manager interviewing a job candidate. There aren’t any hard-and-fast rules about who will be a good hire. You have to consider many factors and rely on your judgment or on a gut feeling based on your experience with people in the workplace. Unfortunately, there’s a great deal of evidence showing that humans have a harder time learning how to integrate information in this way, because they seek rules even when there are none.

In spite of how much it’s part of our nature, we have to overcome the desire for easy answers. No matter how many jumps we make, the magic recipe will not be found:

Ignore that last guy 😉

Institutional Amnesia, Cargo Cults and Software Development

When George Santayana stated that “Those who cannot remember the past are condemned to repeat it.”, he wasn’t talking about technology. When Brenda Michelson and Ed Featherston said much the same thing recently, they were:

https://twitter.com/jetpack/status/573850405026861056
https://twitter.com/jetpack/status/573851102808010752

It’s a sad fact of life that today’s silver bullet is likely to be yesterday’s junk which was probably the day before yesterday’s silver bullet.

Poor design choices are made for a variety of reasons. Sometimes it’s a matter of ego. Sometimes inadequate analysis is the culprit. Focusing on technology rather than problem-solving can be another pitfall. Even attempts at post-hoc justification of a prior bad decision can drive new mistakes.

An uncritical acceptance of tradition is a significant source of problem designs. Eberhard Wolff recently took a swipe at one old standard:

The stock reason for a tiered/distributed design is scalability. However, it’s not a given that distributing X horizontal layers across Y machines (yielding X/Y instances) will yield better results than Y machines, each with all three layers deployed on the same machine. The context in which this sort of distribution makes sense is far from universal. Even when the costs of distribution are outweighed by the benefits, traditional monolithic horizontal layers will likely be less efficient than vertical slices. One of the purported benefits of microservices is the ability to independently scale according to business concerns (vertical slices organized around bounded contexts) rather technology concerns (horizontal layers).

The mention of microservices brings to mind the problem of jumping on bandwagons. How many applications currently under development are being designed using this architectural style because it’s the “next big thing” rather than because the style fits the problem? Sam Newman, author of O’Reilly’s Building Microservices, in “Microservices for Greenfield?”, even states that he considers the style to be more suitable for evolving an existing system rather than building from scratch:

I remain convinced that it is much easier to partition an existing, “brownfield” system than to do so up front with a new, greenfield system. You have more to work with. You have code you can examine, you can speak to people who use and maintain the system. You also know what ‘good’ looks like – you have a working system to change, making it easier for you to know when you may have got something wrong or been too aggressive in your decision making process.

You also have a system that is actually running. You understand how it operates, how it behaves in production. Decomposition into microservices can cause some nasty performance issues for example, but with a brownfield system you have a chance to establish a healthy baseline before making potentially performance-impacting changes.

I’m certainly not saying ‘never do microservices for greenfield’, but I am saying that the factors above lead me to conclude that you should be cautious. Only split around those boundaries that are very clear at the beginning, and keep the rest on the more monolithic side. This will also give you time to assess how how mature you are from an operational point of view – if you struggle to manage two services, managing 10 is going to be difficult.

This same over-eagerness is present in front-end development as much as back-end development. Stefan Tilkow recently tweeted regarding the trend of jumping straight into complex Javascript framework applications rather than evolving into them based on need:

In my opinion, the key to effective design is being able to give a good answer when asked “why”. Being able to articulate the reasons behind the choices made is critical to justifying them. By reasons, I mean a logical explanations of how the techniques chosen contribute to the desired ends. Neither “X recommends this” nor “This is what everybody’s doing” count. Designing, developing, and evolving software systems is not a game of following a recipe. In the words of Grady Booch:

Are Microservices the Next Big Thing?

Bandwagon

The question popped up in a LinkedIn discussion of my last post:

“A question for you, are microservices the next big thing?”

It was a refreshing change of pace to actually be asked about my opinion rather than be told what it is. The backlash against microservices is such that, anything less that outright condemnation is seen as “touting” the latest fad (this in spite of having authored posts such as “Microservice Architectures aren’t for Everyone”, “Microservice Mistakes – Complexity as a Service”, and “Microservices – The Too Good to be True Parts”). Reflex reactions like this will most likely be so simplistic as to be useless, regardless of whether pro or con.

Architectural design is about making decisions. Good design, in my opinion, is about making decisions justified by the context at hand rather than following a recipe. It’s about understanding the situation and applying principles rather than blindly replicating a pattern. No one cares about how cutting edge the technology is, they care that the solution solves the problem without bankrupting them.

So, to return to the question of whether I think microservices are “the next big thing”:

They shouldn’t be, but they might be. Shouldn’t, because a full-blown microservice architecture app, while perfectly appropriate for Netflix, isn’t likely to be appropriate for most in-house corporate applications. They won’t get a benefit because of the additional operational and development complexity. That being said, lots of inappropriate SOA initiatives sucked up a lot of money not that long ago – if software development had a mascot it would be an immortal lemming with amnesia.

The principles behind MSA, however, have some value IMHO. When a monolithic architecture begins to get in the way, those principles can provide some guidance on how to carve it up. The wonderful thing about SRP is its fractal nature, so you split out responsibilities at a level of granularity that’s appropriate for your application’s situation when it makes sense. There’s no rule that you have to carve up everything at once or that you have to slice it as thin as they do at Netflix.

That’s why my posts on the subject tend to sway back and forth. It’s not a recipe for the “right way” (as if there could be one right way) to design applications, it’s merely another set of ideas that, depending on the context, could help or hurt.

It’s not the technique itself that makes or breaks a design, it’s how applicable the technique is to problem at hand.

One Weird Trick to Design Perfect Applications (?)

What’s the right way to design an application?

Scanning the web, one might be tempted to believe that all sites must use AngularJS Backbone.js Ember.js some sort of javascript framework, all services must be RESTful, and if those services aren’t of the “micro” variety, well…

Is that really the case, though?

Is there a right way, or is it case of choosing the way which most closely matches our current context? We can say rules are rules, but experience will teach that rules lose meaning when divorced from the context they were formed in response to. There is no universal best practice. Without understanding the “why” behind a rule, you can’t determine whether it applies.

A conversation with John Evdemon captured principle this in regards to the latest application architecture “sensation” (AKA microservices):



A Kindle highlight shared by Tony DaSilva extends this principle to enterprise architecture:

There is no evidence that structure in and of itself affects the profitability of a company; different structures work best for different companies.

Meeting a need involves two architectures: the architecture of the problem and the architecture of the solution. Understanding the architecture of the problem is a necessary prerequisite to designing the architecture of the solution. Without that context providing definition of the desired end, imperfect though it may be, any proposed solution becomes a shot in the dark. Attempting to extend a technology or technique outside its range of utility risks harming its credibility (see SOA).

Thinking isn’t an option, it’s a requirement.

Microservices – The Too Good to be True Parts

Label for Clark Stanley's Snake Oil Liniment

Over the last several months, I’ve written several posts about microservices. My attitude toward this architectural style is one of guarded optimism. I consider the “purist” version of it to be overkill for most applications (are you really creating something Netflix-scale?), but see a lot of valuable ideas developing out of it. Smaller, focused, service-enabled applications are, in my opinion, an excellent way to increase systems agility. Where the benefits outweigh the costs and you’ve done your homework, systems of systems make sense.

However, the history of Service-Oriented Architecture (SOA), is instructive. A tried and true method of discrediting yourself is to over-promise and under-deliver. Microservice architectures, as the latest hot topic, currently receive a lot of uncritical press, just as SOA did a few years back. An article on ZDNet, “How Nike thinks about app development: Lots of micro services”, illustrates this (emphasis is mine):

Nike is breaking down all the parts of its apps to crate (sic) building blocks that can be reused and tweaked as needed. There’s also a redundancy benefit: Should one micro service fail the other ones will work in the app.

Reuse and agility tend to be antagonists. The governance needed to promote reuse impedes agility. Distribution increase complexity on its own; reuse adds additional complexity. This complexity comes not only from communication issues but also from coordination and coupling. Rationalization, reuse and the ability to compose applications from the individual service is absolutely a feature of this style. The catch is the cost involved in achieving it.

A naive reading of Nike’s strategy would imply that breaking everything up “auto-magically” yields reuse and agility. Without an intentional design, this is very unlikely. Cohesion of the individual services, rather than their size is the important factor in achieving those goals. As Stefan Tilkov notes in “How Small Should Your Microservice Be?”:

In other words, I think it’s not a goal to make your services as small as possible. Doing so would mean you view the separation into individual, stand-alone services as your only structuring mechanism, while it should be only one of many.

Redundancy and resilience are likewise issues that need careful consideration. The quote from the Nike article might lead you to believe that resilience and redundancy are a by-product of deploying microservices. Far from it. Resilience and distribution are orthogonal concepts; in fact, breaking up a monolith can have a negative impact on resilience if resilience is not specifically accounted for in the design. Coupling, in all its various forms, reduces resilience. Jeppe Cramon, in “SOA: synchronous communication, data ownership and coupling”, has shown that distribution, in and of itself, does not eliminate coupling. This means that “Should one micro service fail the other ones will work in the app” may prove false if the service that fails is coupled with and depended on by other services. Decoupling is unlikely to happen accidentally. Likewise, redundant instances of the same service will do little good if a resource shared by those instances (e.g. the data store) is down.

Even where a full-blown microservice architecture is inappropriate, many of the principles behind the style are useful. Swimming with the tide of Conway’s Law, rather than against it, is more likely to yield successful application architectures and enterprise IT architectures. The coherence that makes it successful is a product of design, however, and not serendipity. Microservices are most definitely not snake oil. Selling the style like it is snake oil is a really bad idea.