Benchmark and Trend Assumptions

Sometimes the counterfactual is not a comparison group at all but a constructed number: a benchmark built from historical spending, trended forward. When that is the case, the assumptions used to build the benchmark do not just influence the answer; they largely are the answer. This is the most technical trap in the course and the one that most often explains dueling conclusions.

The benchmark is the counterfactual

Recall from the finance course how an ACO benchmark is built: historical spending, trended forward, risk-adjusted. That benchmark is an estimate of what the population would have cost, the counterfactual, and savings are measured against it. So every judgment call in building it flows straight into the reported savings.

• Pick a higher trend and the benchmark rises, making almost any actual spending look like savings.
• Pick a lower trend and the same actual spending becomes a loss.
• Choose a different baseline period (which historical years to average) and the whole comparison shifts.

Worth remembering: when savings are measured against a constructed benchmark, you are not really measuring the program. You are measuring the program plus a set of assumptions. Change the assumptions and the savings change, without a single patient being treated differently.

Why this produces honest disagreement

This is not usually fraud. Two competent analysts can choose different, defensible trend assumptions and reach opposite conclusions about the same program. CMS’s own benchmark methods have been revised repeatedly precisely because there is no single correct way to build the counterfactual. The ratchet effect from earlier courses lives here too: benchmarks rebased on a program’s own success make future savings mechanically harder, which can turn a working program into an apparent failure over time.

How to read a benchmark-based result

You cannot verify every assumption, but you can ask the right questions:

• What trend was assumed, and does it match reality? An implausibly high trend assumption is the classic way to manufacture savings.
• How was the baseline chosen? Cherry-picked base years bias the result.
• Was the benchmark rebased on the program’s own performance? If so, the ratchet may be hiding a real effect.
• Does an independent comparison group agree? Results that hold up against an observed comparison group are more trustworthy than those resting on a modeled benchmark alone.

Key takeaways

• When the counterfactual is a constructed benchmark, its trend and baseline assumptions largely determine the measured savings.
• Reasonable analysts choosing different assumptions can honestly reach opposite conclusions.
• Interrogate the trend, the baseline, and any rebasing, and trust results more when an observed comparison group agrees.