Measuring Decision Quality: Why Outcomes Aren't Enough
A good decision can produce a bad outcome, and a bad decision can produce a good one—yet most organizations measure only the latter.
This asymmetry creates a systematic blind spot in how we evaluate leadership, strategy, and organizational learning. We celebrate the executive whose risky bet paid off while quietly burying the one whose cautious choice happened to fail. Both deserve scrutiny, but for opposite reasons. The first may have simply gotten lucky. The second may have made the right call given the information available.
The confusion between decision quality and outcome quality is not academic. It shapes how we train leaders, allocate resources, and build institutional memory. When outcomes are the only metric that matters, organizations become outcome-obsessed rather than decision-disciplined. People learn to hide uncertainty, overstate confidence, and cherry-pick evidence. They become skilled at narrative retrofitting—explaining why a poor outcome was actually inevitable, or why a good one was their strategic genius all along.
What gets measured gets managed, and what gets managed gets repeated. If we measure only outcomes, we repeat only the decisions that happened to work, not the ones that were sound.
The thing everyone gets wrong is that decision quality should be evaluated at the moment of decision, not years later when the world has moved on.
This requires a different framework entirely. A high-quality decision is one that:
- Incorporates the best available information at the time it was made
- Acknowledges and quantifies the uncertainties involved
- Follows a defensible process, not intuition or politics
- Considers multiple scenarios and their probabilities
- Builds in mechanisms to test assumptions and adjust course
Consider a pharmaceutical company deciding whether to advance a drug candidate to Phase III trials. The decision quality depends on the strength of Phase II data, the competitive landscape, the regulatory pathway, and the company's risk tolerance. It does not depend on whether the drug ultimately succeeds—an outcome influenced by thousands of variables beyond the company's control, including regulatory decisions, manufacturing challenges, and market adoption patterns that couldn't be predicted.
Yet most companies evaluate this decision retrospectively. If the drug succeeds, the decision-maker is promoted. If it fails, they're questioned. The actual quality of the reasoning—whether it was sound given what was known—becomes invisible.
Why this matters more than people realize is that outcome-based evaluation systematically selects for the wrong behaviors.
It rewards overconfidence. It punishes appropriate caution. It creates perverse incentives to take on hidden risks that might pay off spectacularly or fail catastrophically, but rarely in the middle. Over time, organizations staffed by people selected for lucky outcomes rather than sound reasoning become fragile. They've optimized for a narrow band of past conditions rather than for robustness across scenarios.
The alternative is to measure decision quality directly. This means documenting the reasoning before the outcome is known. It means assigning probabilities to scenarios and comparing them to what actually occurred. It means asking: "Given what we knew then, was this the right call?" separately from "Did it work out?"
This requires discipline. It requires admitting uncertainty rather than performing certainty. It requires organizations to build institutional structures—decision logs, pre-mortems, scenario planning, probability calibration exercises—that make reasoning visible and comparable over time.
The payoff is an organization that learns from experience rather than merely surviving it. Teams that can distinguish between good decisions that failed and bad decisions that succeeded will make better decisions going forward. They'll take appropriate risks rather than either reckless or timid ones. They'll build institutional knowledge rather than institutional mythology.
The best organizations don't measure outcomes. They measure the quality of reasoning that produced them. Everything else follows.