Module 1
The Evaluation Problem
Why measuring whether a program worked is genuinely hard: the counterfactual, confounding, and what 'worked' even means.
By the end of this module, you will be able to:
- Explain why every claimed effect rests on an unobservable counterfactual
- Identify confounding in a comparison between program participants and non-participants
- Specify the aim and the perspective implied by any claim that a model worked
- The Counterfactual Every evaluation is a comparison against something that never happened. Understanding that impossible comparison is the whole game. About 5 min
- Correlation Is Not Causation The oldest warning in statistics is the one most often ignored in health policy. This lesson covers confounding and why it fools smart people. About 5 min
- What 'Worked' Means A model can succeed and fail at the same time, depending on the question. This lesson pins down what you are actually asking before you read an answer. About 4 min
Module quiz
Answer all questions to see your score.
1. What is the 'counterfactual' in an evaluation?
Evaluation compares what happened to what would have happened otherwise; that unobservable alternative is the counterfactual, and it must be estimated.
2. Program participants had lower costs than non-participants. Why might that not prove the program worked?
Voluntary joiners often differ from non-joiners before the program starts, so the raw comparison confounds selection with the program's effect.
3. A model improves quality but produces no net savings. Did it 'work'?
'Worked' is undefined until you specify the aim (quality, cost, outcomes, equity) and whose ledger you are reading.