Back to Module 1: The Evaluation Problem

Lesson 3

What 'Worked' Means

About 4 min

A model can succeed and fail at the same time, depending on the question. This lesson pins down what you are actually asking before you read an answer.

Before you can judge whether a program worked, you have to answer a prior question that is skipped surprisingly often: worked at what, and for whom? A model can succeed and fail in the same evaluation, and treating “did it work” as a single yes-or-no question is how people talk past each other about the evidence.

Worked at what

The four aims from the intro course are four different questions, and a model can answer them differently:

  • Did it improve quality?
  • Did it improve outcomes?
  • Did it reduce cost, and cost to whom?
  • Did it advance equity?

CPC+, the primary care model from the case-studies module, is the cleanest illustration: it reduced emergency visits and hospitalizations but produced no net savings. Was that a success? On utilization and care delivery, yes. On the taxpayer’s ledger, no. Both statements are true, and only naming the aim resolves the apparent contradiction.

Worked for whom

“Saved money” also hides a question: saved money for whom? A model can lower a health plan’s spending while shifting cost to patients, or lower provider spending while raising federal outlays. The finance course’s insistence on total cost of care was really an insistence on being clear about whose ledger you are reading.

Perspective”Worked” might mean
Taxpayer / MedicareNet federal savings after program costs
Health planLower plan spending
ProviderPositive margin under the contract
PatientBetter outcomes, lower out-of-pocket cost

Worth remembering: never accept an unqualified “it worked” or “it failed.” Ask: worked on which aim, from whose perspective, over what timeframe? Most disagreements about value-based care evidence dissolve once those three things are specified.

Why this comes first

This is the setup for the whole course. Every later lesson, comparison groups, regression to the mean, gross versus net savings, is a tool for answering a specific question well. If you have not pinned down the question, the most rigorous method in the world will give you a precise answer to the wrong thing.

Key takeaways

  • “Worked” is undefined until you name the aim (quality, outcomes, cost, equity) and the perspective (taxpayer, plan, provider, patient).
  • A model can genuinely succeed on one aim and fail on another, as CPC+ did.
  • Specify the question first; method only matters once you know what you are asking.

Check your understanding

A model measurably improves quality and reduces hospitalizations but produces no net savings. Did it 'work'?

Share