It’s ironic that the traits we think of as making a good developer are also those that can get in the way of design and testing, but that’s just the case. Think of how many times you’ve heard (or perhaps, said) “no one would ever do that”. Yet, given the event-driven, non-linear nature of modern systems, if a given execution path can occur, it will occur. Our cognitive biases can blind us to potential issues that arise when our product is used in ways we did not intend. As Thomas Wendt observed in “The Broken Worldview of Experience Design”:
To a certain extent, the designer’s intent is irrelevant once the product launches. That is, intent can drive the design process, but that’s not the interesting part; the ways in which users adopt the product to their own needs is where the most insight comes from. Designer intent is a theoretical, speculative formulation even when based on the most rigorous research methods and valid interpretations. That is not to say intention and strategic positioning is not important, but simply that we need to consider more than idealized outcomes.
Abhi Rele, in “APIs and Data: Journey to the Center of the Customer Experience”, put it in more concrete terms:
If you think you’re in full control of your customers’ experience, you’re wrong.
Customers increasingly have taken charge—they know what they want, when they want it, and how they want it. They are using their mobile phones more often for an ever-growing list of tasks—be it searching for information, looking up directions, or buying products. According to Google, 34% of consumers turn to the device that’s closest to them. More often than not, they’re switching from one channel or device mid-transaction; Google found that 67% of consumers do just that. They might start their product research on the web, but complete the purchase on a smartphone.
Switch device in mid-transaction? No one would ever do that! Oops.
We could, of course, decide to block those paths that we don’t consider “reasonable” (as opposed to stopping actual error conditions). The problem with that approach, is that our definition of “reasonable” may conflict with the customer’s definition. When “conflict” and “customer” are in the same sentence, there’s generally a problem.
These conflicts, in the right domain, can even have deadly results. While investigating the Asiana Airlines crash from July of 2013, one of the findings of the National Transportation Safety Board (NTSB) was that the crew’s belief of what the autopilot system would do did not coincide with what it actually did (my emphasis):
The NTSB found that the pilots had “misconceptions” about the plane’s autopilot systems, specifically what the autothrottle would do in the event that the plane’s airspeed got too low.
In the setting that the autopilot was in at the time of the accident, the autothrottles that are used to maintain specific airspeeds, much like cruise control in a car, were not programmed to wake up and intervene by adding power if the plane got too slow. The pilots believed otherwise, in part because in other autopilot modes on the Boeing 777, the autothrottles would in fact do this.
“NTSB Blames Pilots in July 2013 Asiana Airlines Crash” on Mashable.com
Even if it doesn’t contribute to a tragedy, a poor user experience (inconsistent, unstable, or overly restrictive) can lead to unintended consequences, customer dissatisfaction, or both. Basing that user experience on assumptions instead of research and/or testing increases the risk. As I’ve stated previously, risky assumptions are an assumption of risk.
Form Follows Function 
Good post. Two things come to my mind:
1) Read this as I was just done teaching a course on model-driven architecting & UML. The 10 participants were 5 experienced engineers, of which 3 very experienced, and 5 juniors. At the end of the three-day course I gave them an assignment, an imaginary extension of functionality for the Twittter platform. It was amazing how the juniors rushed headlong into getting a demonstrator, then a prototype up and running – and how the seniors took time to discuss the requirements, think, and then came up with hidden assumptions and unforeseen consequences of the same.
2) On a related note: I used to work in the aerospace sector, where it has been known for years and years, actually since mid-1990s, that “pilot confusion” is an important contributing factor in aircraft crashes. “Pilot confusion” in the sense that a pilot’s mind may be driven into a cognitively incoherent state ( also called “doubt” or “confusion” ) by not being certain about what mode the autopilot is in. Companies like Rockwell, who produce avionics, repeatedly promised they would factor this into the design of newer autopilot generations. Airbus has even gone as far as engaging psychologists to diagnose, live, pilot confusion in simulator runs. As Boeing, however, had already taken fundamental design decisions w/ regard to the 777 by mid-1990s. This means that current 777 avionics still suffer from both a lack of knowledge and insight and flawed assumptions that go back to the 1980s, and beyond.
LikeLike
Thanks Jan.
Regarding your first point – yes, I always remember the old saying “talk is cheap”. There is some value in using a mockup to visualize things, but jumping right into code leaves you vulnerable to the sunk cost fallacy I highlighted in my previous post – “I’ve put all this effort into this, I can’t just throw it away”. Experience can well be a factor in knowing how much up front analysis to put in, as well as what kind of pitfalls to look for.
The second point is a little scary – “pilot” and “confusion” are two more words that really don’t belong in the same sentence, let alone adjacent. That being said, I’d imagine it’s far from a simple problem.
LikeLike
@Gene I do agree about the second point being a little scary. Just google “pilot confusion”. You’ll be amazed – and scared. When that’s over, just keep a tally of how often pilot confusion is mentioned in connection with Boeing aircraft, and how often with Airbus aircraft. You’ll see a net predominance of the Boeing case.
LikeLike
Pingback: Design Follies – Architect Knows Best | Form Follows Function
Pingback: Design Communicates the Solution to a Problem | Form Follows Function
Pingback: Fixing IT – Credible or Cassandra? | Form Follows Function
Pingback: Problem Space, Solution Space, and Tunnel Vision | Form Follows Function