The Counterfactual

Every claim that a value-based program “saved money” or “improved outcomes” rests on a comparison, usually an invisible one. It compares what actually happened to what would have happened without the program. That second thing never occurred, cannot be observed, and yet the entire evaluation depends on estimating it. This is the counterfactual, and it is the foundation of everything in this course.

The problem in one sentence

You cannot rerun history. A group of patients lived through a program; you will never see how that exact group would have fared without it. So the honest question is never “what happened?” (that you can measure) but “what happened compared to what otherwise would have?” (that you must estimate).

Worth remembering: measuring what happened is easy. Estimating what would have happened otherwise is the hard part, and it is where every evaluation succeeds or fails. A number without a credible counterfactual is not evidence of an effect; it is just a number.

Why “before and after” is not enough

The tempting shortcut is to compare a program’s population before and after: spending was $9,000 per member, now it is $8,700, so the program saved $300. But costs move for a hundred reasons, a mild flu season, a new generic drug, an economic shift. Some of that $300 might have happened anyway. Without an estimate of the counterfactual, you cannot separate the program’s effect from everything else that changed. The next module is entirely about how to build that estimate.

You have already met the counterfactual

The finance course called it something else: the benchmark. An ACO benchmark is an estimate of what the population would have cost without the ACO. A bundle’s target price is the counterfactual for an episode. Every one of those numbers is an attempt to answer the same impossible question, which is why the assumptions inside them decide the result. Evaluation and payment are two faces of the same problem.

Why this matters for a reader of evidence

When you read that a model produced savings, your first question should be: compared to what? How was the counterfactual estimated, and is that estimate credible? A study that cannot answer that question clearly has not shown an effect, no matter how large its headline number. Training yourself to ask “compared to what?” before anything else is the single most valuable habit this course can give you.

Key takeaways

• Evaluation compares what happened to what would have happened otherwise: the counterfactual.
• The counterfactual is unobservable, so it must be estimated, and that estimate is where evaluations live or die.
• Benchmarks and target prices are counterfactual estimates too; always ask “compared to what?”