Form Follows Function on SPaMCast 369


This week’s episode of Tom Cagley’s Software Process and Measurement (SPaMCast) podcast, number 369, features Tom’s essay on stand-up meetings, Kim Pries on mastery, and a Form Follows Function installment on #NoEstimates.

Tom and I discuss my post “#NoEstimates – Questions, Answers, and Credibility” and take on whether it’s realistic to eliminate estimates given their value to customers.

What We Have Here is a Failure to Communicate

“What we’ve got here is failure to communicate” and it appears to be epidemic. My own personal grand unified theory of everything is that most problems stem from or are aggravated by a lack of communication. Whether the topic is process, governance, planning, estimates, or design, chances are it’s easier to find opinions (and worse, policy and practices) based on one-sided viewpoints than a balanced understanding of the contexts involved. This is dangerous due to the simple fact that organizations are social systems (frequently fractal systems of systems) and as Ruth Malan has noted:

Russell Ackoff urged that to design a system, it must be seen in the context of the larger system of which it is part. Any system functions in a larger system (various larger systems, for that matter), and the boundaries of the system — its interaction surfaces and the capabilities it offers — are design negotiations. That is, they entail making decisions with trade-off spaces, with implications and consequences for the system and its containing system of systems. We must architect across the boundaries, not just up to the boundaries. If we don’t, we naively accept some conception of the system boundary and the consequent constraints both on the system and on its containing systems (of systems) will become clear later. But by then much of the cast will have set. Relationships and expectations, dependencies and interdependencies will create inertia. Costs of change will be higher, perhaps too high.

In other words, systems exist within an ecosystem, not a vacuum. Failure to take context into account harms systems (whether software or social) by baking in harmful structures and behaviors and we cannot take into account contexts that are not communicated and appreciated. This, by the way, is why you find posts about management and process on a site with the tagline “All Things Architectural”.

This is why I believe that successfully managing technical debt can’t happen without successfully communicating to the customer when it’s being taken on, what the costs involved are (or may be) and how it’s affecting the evolution of the product.

This is why I believe the answer to problems related to estimation lies in communication and collaboration, rather than #NoEstimates on the one hand or rigid authoritarianism on the other. This, in my opinion, holds true for all the social system issues (management, process, governance, planning, quality, and architectural design) that affect software development. Without understanding (which does not happen without communication) the goals behind the practices and what results are being achieved, it’s unlikely that the system will work to the satisfaction of anyone.

Local “optimizations” won’t fix systemic problems. We need to bridge the gaps. Can we talk?

Updated 4/8/2016 to fix a broken link.

Conflicts, Collaboration, and Customers

With Ron Jeffries comment on my “Negotiating Estimates” post, I think we’ve reached agreement on a common problem, even if we disagree (perhaps?) over how to fix it. As I noted in my short answer, my position is that abuse of estimates stems from a deeper issue. Changing a practice won’t eliminate the abuse. Changing culture will. Changing that culture (where possible) involves convincing the players that it’s in their best interests to change, which it is.

This is my biggest objection to #NoEstimates. It makes an illogical leap that the practice is responsible for problems rather than some underlying dysfunction without showing causality. This putting the cart before the horse risks encouraging people to espouse action that is probably both ineffective and detrimental to their relationship with their customer. The likeliest outcome I see from that is more problems, not fewer.

The word “customer” was a point of contention:

It’s my position that, for better or for worse, there is a customer/provider relationship. Rather than attempt to deny that, it makes more sense, in my opinion, to concentrate on having the best possible customer/provider relationship. While poor customer service is extremely common, I’d hope everyone can think of at least one person they enjoy doing business with, where the relationship is mutually beneficial. I know I certainly can, which stands as my (admittedly unscientific) evidence that the relationship need not be adversarial. A collaborative, mutually beneficial relationship works out best for both parties.

“Collaborative”, however, does not mean conflict free. In his post “Against Estimate-Commitment”, Jeffries states:

The Estimate-Commitment relationship stands in opposition to collaboration. It works against collaboration. It supports conflict, not teamwork.

I both agree and disagree.

There will be conflict. Different team members will have different wants. Different stakeholders will have different wants. The key is not in preventing conflict, but resolving it in a way agreeable to all parties:

Recognizing, rather than repressing conflicts might even be beneficial:

What is critical, is that the relationship between the parties not be one-sided (or even be perceived as one-sided). In-house IT often has the peculiar characteristic of seeming one-sided to both customers and providers. The classic model of IT provision is, in my opinion, largely to blame. It seems almost designed to put both parties at odds.

The problem can be fixed, however, with cultural changes that encompass both IT and the business units it serves. One such change is moving from a project-centric to a product-centric model that aligns the interests of both groups. This alignment cements the relationship, making IT a valuable partner in the process rather than an obstacle to be overcome to get what’s wanted/needed. Relationships are key to success. By structuring the system so that each group’s incentives, conflicts and all, are aligned to a common goal, the system can be made to work. We’ve certainly had plenty of examples of what we get from the opposite situation.

Negotiating Estimates

Congress of Vienna

In my previous post dealing with Ron Jeffries (since revised) “Summing up the discussion”, I focused strictly on the customer-focused aspects. I did, however, note some language regarding negotiating estimates that I wanted to touch on:

“Negotiating” estimates is deeply embedded into most cultures. It probably started in the marketplace in the village in ancient Greece, where the carrot guy tried to get three hemitetartemorions for his carrots, and your great-to-the-nth grandmother talked him down to one.
We assume that a contractor’s estimate has fat in it and we assume that we need tough negotiation to squeeze it out. The better the contractor is at estimating, the more this process hurts him, because he has nothing left to squeeze. And in the end, it hurts the buyer as well: the only way the contractor has to survive is to cut quality.

Jeffries is absolutely correct that negotiation is an ancient tradition. Likewise, he is correct that shaving an accurate estimate may well end up costing the customer in reduced quality (especially if giving point estimates, which is an incredibly poor practice, in my opinion). What he fails to mention, however, is that negotiating an estimate need not happen. In fact, negotiating an estimate without negotiating the deliverable is pretty much the worst possible thing you could do. You’re risking either your profit margin or any future business with the customer and quite possibly, both. It’s a bad idea for a vendor and incredibly ill-considered for in-house IT (it’s not like you can take the money and run).

In my opinion, when someone is willing to change an estimate without a change in what they’re estimating, it’s a bad sign. If you adequately understand the information at hand, have some experience, and have made a good faith effort, there’s no reason to be willing to change the estimate without learning more about what is or is not needed. It’s not really a negotiation if only one party is getting what they want, particularly if the other party is getting abused. For the abused party, striking back (by shaving quality without the customer’s knowledge) is counter-productive. The only negotiation should be what’s in scope and what’s not.

In a balanced relationship, the provider can explain what the customer is getting for their money and the customer realizes they won’t get something for nothing. Communication and collaboration can provide the basis for trust. Trust is essential for both parties to become partners in delivering the best possible product.

‘Customer collaboration over contract negotiation’ and #NoEstimates

Faust and Mephistopheles

Since my last post on #NoEstimates, the conversation has taken an interesting turn. Steve McConnell weighed in on July 30th, with a response from Ron Jeffries on the following day. There have been several posts from each since, with the latest from Jeffries titled “Summing up the discussion”. In that post, he states the following (emphasis added):

The core of our disagreement here is that Steve says repeatedly that if the customer wants estimates, then we should give them estimates. He offers the Manifesto’s “Customer collaboration over contract negotiation”1 as showing that we should do that. (It’s interesting that he makes that point in a kitchen where surely he expects that the couple and the contractor(!) are going to negotiate a contract.)

To anyone who has been even minimally involved with business, the idea that a contract will be negotiated should not a surprise, no matter how agile the provider. In spite of his footnote warning “I confess that I am singularly unamused when people bludgeon me with quotes from the Manifesto”, I have to note that the Agile Manifesto does say “…while there is value in the items on the right, we value the items on the left more” rather than “only the items on the left have value”.

In the post, Jeffries says that in an agile transformation, the “…contractor-customer relationship gets inverted”. I have to confess that the statement is a bit baffling. A situation where “the customer is always right” can certainly degrade into a no-win scenario. Inverting that relationship, however, is equally bad. Good, sustainable business relationships should be relatively (even if not perfectly) balanced. Moving away from minutely detailed contracts that prevent any change without days of negotiation just makes sense for both parties. Eliminating all obligation on the part of the provider, however, is a much harder sell.

Jeffries asserts that “Agile, if you’ll actually do it, changes the fundamental relationship from an estimate-commitment model to a truly collaborative stream of building up the product incrementally.” If, however, you fail to meet your customer’s needs (and, as I pointed out in “#NoEstimates – Questions, Answers, and Credibility”, one of the best lists of why estimates are needed for decision-making comes from Woody Zuill himself), how collaborative is that? Providing estimates with the caveat that they are subject to change as the knowledge about the work changes is more realistic and balanced for both parties. That balance, in my opinion (and experience) is far more likely to result in a collaboration between customer and provider than any inversion of the relationship.

Trust, is essential to fostering collaboration. If we fail to understand our customers’ needs (which impression is given when we make statements like that in the tweet below), we risk fatally damaging the trust we need to create an effective collaboration.

Estimates – Uses, Abuses, and that Credibility Thing Again

In my last post, “#NoEstimates – Questions, Answers, and Credibility”, I brought up the potential for damaging ones credibility by proposing changes without being able to explain why the change should be beneficial. Consider the following exchange on Twitter:

When the same person states “There is a lot written about this” and “The question is young and exploration ongoing” back to back, it is reasonable to wonder how much thought has been put into the “question” versus the desired solution. It’s also reasonable to wonder if this really qualifies as a “question”:

Some people latch on to ideas that tell them they shouldn’t have to do things they don’t like to do. Responsibility, however, sometimes requires us to do things we don’t like to do. It’s not a matter of being tied to a particular practice, but a matter of avoiding things that make us look foolish such as these:

[Note: I find these dismaying not because of opposition to agility, but because I am a proponent of agility.]

None of this is to deny that some misuse, even abuse estimates. That being said, abusive managers/clients are unlikely to either surrender their “stick”. The problem is not the tool, but how it’s employed. It might be useful to consider exactly how likely they are to give up being abusive even if they do agree to quit asking for estimates.

Many of the issues around estimates stem from communication and knowledge issues in my opinion. Typically, the less we know about something, the harder it is to reason about it (size, complexity, etc.). In my experience, two practices mitigate this: providing estimates as a range (wider initially, with a narrower range when the item is better defined) rather than a single point and changing the estimate when new information arises (regardless of whether the new information is an unforeseen issue found by the team or a change requested by the customer). It shouldn’t be a surprise that it’s hard to succeed by making a single estimate up front and refusing to revise it no matter what new information comes in. That’s a bit like driving somewhere by putting on a blindfold and following the GPS without question.

Another practice that can help with uncertainty is using a proof of concept. When in unknown territory, I’ve found it’s easier to estimate how long to explore something than to estimate both the exploration and the implementation of a specific feature. Additionally, this practice makes it explicit for the customer that they’re paying for the learning, not feature itself. Transparency about the risks allows them to accurately determine how much they’re willing to pay just to find out if the tool/technology/etc. will meet their needs.

This is probably a good time to repeat the way estimates are used (from Woody Zuill’s “Why do we need estimates?”):

  • To decide if we can do a project affordably enough to make a profit.
  • To decide what work we think we can do during “a Sprint”. (A decision about amount)
  • To decide what work we should try to do during “a Sprint”. (A decision about priority or importance)
  • To decide which is more valuable to us: this story or that story.
  • To decide what project we should do: Project A or Project B.
  • To decide how many developers/people to hire and how fast to “ramp up”.
  • To decide how much money we’ll need to staff a team for a year.
  • To give a price, or an approximate cost so a customer can decide to hire us to do their project.
  • So we can determine the team’s velocity.
  • So marketing can do whatever it is they do that requires they know 6 months in advance exactly what will be in our product.
  • Someone told me to make an estimate, I don’t use them for anything.

All of these are valid enough as stated, but let’s re-word a couple

  • To help the customer decide what work we should try to do during “a Sprint”. (A decision about priority or importance)
  • To help the customer decide which is more valuable to us: this story or that story.
  • To help the customer decide what project we should do: Project A or Project B.
  • To help the customer decide how much money we’ll need to staff a team for a year.

It needs to be really clear that the customer’s needs are the important ones here.

Another use is to help determine whether a particular issue is worth addressing. It’s hard to determine the level of technical debt without a sense of what the cost is to fix versus the cost of not fixing it.

To determine value, we need to know both benefit and cost. A feature that earns a million dollars for a cost of $999,999 is a lousy deal compared to one making $10,000 for a cost of $1,000.

Shaping our practices around our customer’s needs is a good way to create a partnership with those customers. In my opinion, that’s an easier sell to the customer than “dump the estimates and let’s see what happens”.

#NoEstimates – Questions, Answers, and Credibility

This is questioning?

A recent episode of Thomas Cagley’s SPAMCast featured Woody Zuill discussing #NoEstimates. During the episode, Woody talked about his doubts about the usefulness of estimating and the value of questioning processes rather than accepting them blindly.

I’m definitely a fan of pragmatism. Rather than relying on dogma, I believe we should be constantly evaluating our practices and improving those found wanting. That being said, to effectively evaluate our practices, we need to be able to frame the evaluation. It’s not enough to pronounce something is useful or not, we have to be able to say who is involved, what they are seeking, what the impact is on them, and to what extent the outcome matches up with what they are seeking. Additionally, being able to reason about why a particular practice tends to generate a particular outcome is critical to determining what corrective action will work. In short, if we don’t know what the destination looks like, we will have a hard time steering toward it.

In his blog post “Why do we need estimates?”, Woody (rightly, in my opinion) identifies estimates as a tool used for decision-making. He also lists a number of decisions that estimates can be used to make:

  • To decide if we can do a project affordably enough to make a profit.
  • To decide what work we think we can do during “a Sprint”. (A decision about amount)
  • To decide what work we should try to do during “a Sprint”. (A decision about priority or importance)
  • To decide which is more valuable to us: this story or that story.
  • To decide what project we should do: Project A or Project B.
  • To decide how many developers/people to hire and how fast to “ramp up”.
  • To decide how much money we’ll need to staff a team for a year.
  • To give a price, or an approximate cost so a customer can decide to hire us to do their project.
  • So we can determine the team’s velocity.
  • So marketing can do whatever it is they do that requires they know 6 months in advance exactly what will be in our product.
  • Someone told me to make an estimate, I don’t use them for anything.

What is missing, however, is an alternate way to make these decisions. It’s also missing from the follow up post “My Customers Need Estimates, What Do I do?”. If the customer has a need, does it seem wise to ask them to abandon (not amend, but abandon) a technique without proposing something else? Even “let’s try x instead of y” is insufficient if we can’t logically explain why we expect “x” to work better than “y”. The issue is one of credibility, a matter of trust.

In her post “What Creates Trust in Your Organization?”, Johanna Rothman related her technique for creating trust:

Since then, I asked my managers, “When do you want to know my project is in trouble? As soon as it I think I’m not going to meet my date; after I do some experiments; or the last possible moment?” I create trust when I ask that question because it shows I’m taking their concerns seriously.

After that project, here is what I did to create trust:

  1. Created a first draft estimate.
  2. Tracked my work so I could show visible progress and what didn’t work.
  3. Delivered often. That is why I like inch-pebbles. Yes, after that project, I often had one- or two-day deliverables.
  4. If I thought I wasn’t going to make it, use the questions above to decide when to say, “I’m in trouble.”
  5. Delivered a working product.

While I can’t say that Johanna’s technique is the optimal one for all situations, I can at least explain why I can put some faith in it. In my experience, transparency, collaboration, and a respect for my stakeholders’ needs tends to work well. Questions without answers? Not so much.