Becoming a Critical Consumer

This course has built a way of reading evidence. This closing lesson turns it into a checklist you can carry into any evaluation, and names the posture that makes the checklist worth having.

The checklist

Read any value-based evaluation against these questions, roughly in order:

1. Compared to what? How was the counterfactual estimated? A comparison group, a constructed benchmark, or just before-and-after? A weak counterfactual sinks everything downstream.

2. Who was in the groups, and were they comparable? Voluntary participation, matching only on the measured, or true randomization? Assume selection until shown otherwise.

3. Could regression to the mean explain it? If the program targeted last year’s most extreme patients, was there a comparison group selected the same way?

4. Who left, and how were they counted? Attrition can manufacture improvement. Look for intent-to-treat analysis.

5. Gross or net? Were shared-savings payouts and program costs subtracted? For a public program, the net-to-taxpayer figure is the one that matters.

6. How uncertain, and how big? Is the effect distinguishable from zero, and is it large enough to matter if real?

7. Would it generalize here? Same population, condition, and era? Or an early-adopter pilot unlikely to survive scale?

The claim	The question it must answer
”Saved money”	Gross or net, and compared to what?
”Reduced costs for high-risk patients”	Versus a similar group, or just regression?
”Participants did better”	Better than a comparable non-participant group?
”The pilot succeeded”	Would it hold at scale, mandatory, everywhere?

The posture

The checklist is a tool; the posture is what makes it honest. A critical consumer of evidence is neither a booster nor a cynic. The booster accepts favorable numbers because they support the cause; the cynic rejects everything because nothing is perfect. Both stop thinking. The analyst’s job is harder: take each result on its methods, state plainly what it does and does not show, and let the aggregate picture, encouraging in places, unproven in others, stand as it is.

Worth remembering: for an institute whose name goes on its research, the discipline in this course is not optional rigor; it is the product. The value of a nonpartisan analysis is precisely that it applies the same scrutiny to results it likes and results it does not. That even-handedness is what makes it worth trusting.

The through-line

Value-based care is a set of hypotheses about how to pay for health. Evaluation is how we learn which ones are true. Done poorly, it manufactures success stories that waste money and trust. Done well, it is the slow, honest process of finding the designs that actually work, and setting aside the ones that only looked like they did. Being a careful reader of that evidence is how you contribute to getting the answer right.

Key takeaways

• Run every evaluation through the checklist: counterfactual, comparability, regression, attrition, gross versus net, uncertainty and magnitude, and generalizability.
• The critical-consumer posture is neither booster nor cynic; judge each result on its methods and report what it honestly shows.
• For a nonpartisan institute, even-handed rigor is the product, and it is what makes the analysis worth trusting.