Why Pre-Post Comparisons Mislead

The most intuitive evaluation is also one of the weakest: measure the program’s population before it starts, measure it after, and call the difference the effect. It feels airtight and it usually is not. Understanding why is the bridge from the counterfactual problem to how evaluators actually solve it.

The background is always moving

A pre-post comparison assumes that nothing except the program changed. Nothing is ever that still. Over any evaluation period:

• Medical prices and drug costs shift.
• New treatments and generics appear.
• The population ages, and its health mix changes.
• The broader economy and health system move.

A program’s spending can fall simply because a blockbuster drug went generic that year. Attribute that to the program and you have credited it with something it did not do. Pre-post cannot tell the two apart, because it has no way to see what the background trend was.

The fix: a comparison group

The solution is to watch a comparison group, a similar population that did not get the program, over the same period. The background trends hit both groups. If the program group improved more than the comparison group, that extra movement is a candidate for the program’s real effect.

Worth remembering: the question is never “did the program group get better?” It is “did the program group get better than it would have anyway?” A comparison group is how you estimate the “anyway.” Without one, you are measuring the program plus the entire rest of the world, and calling the total the program.

A concrete illustration

Suppose a program’s costs rise 3 percent over two years. Success or failure? Unanswerable alone. If a comparable non-program population rose 7 percent over the same two years, the program group’s slower growth suggests a real effect. If the comparison group rose only 1 percent, the program group actually did worse than the background. The same raw number, 3 percent, points in opposite directions depending on the comparison. That is why the comparison group, not the raw change, carries the evidence.

Key takeaways

• Pre-post comparisons assume nothing but the program changed, which is never true.
• Background trends (prices, treatments, demographics) move outcomes independent of any program.
• A comparison group experiencing the same trends is what isolates the program’s added effect.