You Can't Measure Productivity... Accurately

It's not just me saying this. Gergely Orosz and Kent Beck quickly wrote an excellent response article that I recommend to everyone. Martin Fowler also backed this up years ago in a very direct article, Cannot Measure Productivity, and I completely agree with him. There, he states: In particular we have no way of reasonably measuring productivity. — Martin Fowler 2003

The crux of the issue is that no metric can accurately capture the value delivered in software development. Number of features, tickets, releases, deploys, merges, commits, lines of code... They are all imperfect indicators. They are arguable and can be easily gamed. It's too easy to increase the metric without significantly increasing value.

This means that if you put the metric in the spotlight and make it the goal—be it for promotions, bonuses, or redundancy processes—the metric itself becomes the problem to be solved. This takes the focus away from well-intentioned outcomes. People will optimise for the metric rather than for the company's best interest. Employees under pressure to meet these metrics will focus too much on the metrics themselves. The ill-intentioned will game them without hesitation, and the best employees, the ones who genuinely care, will lose morale over the whole situation.

This leads me to the following conclusions around productivity metrics:

• You can't rely solely on them for decision-making.
• You can't put them in the spotlight and make them the goal.

This is commonly illustrated by Goodhart's Law: "Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes."

And there is a whole book full of real-life stories about how making metrics the goal or relying solely on them for decision-making led to undesirable consequences in schools, medical care, businesses, and government: The Tyranny of Metrics.

This is an opinion that is very well established within the industry.

But You Can Measure Anything... If You Actually Wanted To

But what if metrics weren't in the spotlight? What if teams had bigger concerns and viewed metrics merely as a way to reduce uncertainty?

Take, for example, a team concerned about a slow and flaky CI pipeline. If they invest time in improving it, wouldn't they want to know if their deployment frequency increased as a result? Here, deployment frequency is as arguable and gameable as any other metric we've discussed. But now, the team is motivated by the greater good of improving the developer experience. Metrics simply become a way to validate success by measuring outcomes.

By starting from a problem we actually care about, we have a north star that guides our decisions. This allows us to benefit from the additional insights that metrics provide without compromising our intentions.

Engineering teams love to tackle problems and are vocal about them. Want to decrease quality issues? Ask where they're coming from. Want to increase productivity? Ask what's slowing the team down. Start with why. Once you identify a problem the team cares about, they'll happily measure.

But this doesn't resolve McKinsey's problem doesn't it?

McKinsey's problem

McKinsey's article targets C-level executives who constantly need ways to abstract and simplify decision-making in companies with hundreds or thousands of employees. I believe Kent and Gergely are spot-on when discussing the kinds of problems the proposed productivity metrics will be used for: deciding redundancies, raises, and team investments.

Unlike my earlier example, these decision-makers focused on the greater good of company efficiency and profitability are not the same people being measured. And when jobs and salaries are at stake, metrics steal the show and become the focus.

In situations like this, it is critical choosing metrics that accurately represent the greater good. But if these wouldn't exist, we would be inevitably hurting the company. Consider this dramatic example from "The Tyranny of Metrics":_"Numerous studies have shown that when surgeons are rated or remunerated according to their success rates, some respond by refusing to operate on patients with more complex or critical conditions."_ If people are willing to gamble with lives, what do you think they'd do in software delivery?

This is why this article is causing such an uproar. And it is completely understandable. Sending a message like this can cause important damage to companies and individuals.

Personally, I've seen data and metrics used successfully to validate hypotheses, bring visibility to day-to-day operations, raise awareness of problems, track progress toward goals, and validate success. I believe metrics are a great tool. But one we must use with care.

‍