Significance, Uncertainty, and Magnitude

A single savings figure hides the two things that tell you how much to trust it: how uncertain it is, and how big it is. A number without those two companions is half a fact. This lesson is about reading the numbers around the number.

Uncertainty: the confidence interval

Every estimate from a sample carries uncertainty, usually shown as a confidence interval: a range the true effect probably falls within. “Savings of 2 percent (95% CI: -1% to +5%)” means the real effect could plausibly be anywhere from a 1 percent loss to a 5 percent saving. When that range includes zero, the result is not statistically significant: the data cannot rule out no effect at all.

Worth remembering: “not statistically significant” does not mean “the program failed.” It means the study could not distinguish the effect from zero, often because the effect was small, the sample was too small, or both. It is an absence of clear evidence, not evidence of absence.

Magnitude: is the effect big enough to matter?

Statistical significance and practical importance are different questions. With a large enough sample, a trivially small effect can be statistically significant, and a genuinely meaningful effect can miss significance in a small one.

Result	Statistically significant?	Practically meaningful?
0.2% savings, huge sample	Yes	Barely
6% savings, tiny sample	Maybe not	Yes, if real

So you have to read both. A significant 0.3 percent saving may not be worth the program’s disruption. A non-significant 5 percent saving might justify a larger study rather than dismissal.

Underpowered studies

Many value-based evaluations are underpowered: the population is too small, or the effect too modest, for the study to detect a real effect reliably. An underpowered null result is genuinely ambiguous, it is consistent with a real effect the study simply could not see. This is why “no significant effect” from a small pilot should prompt “we need better evidence,” not “it does not work.”

How to read for it

• Find the confidence interval, not just the point estimate. Does it include zero? How wide is it?
• Judge magnitude separately: is the effect, if real, big enough to matter?
• Ask whether the study was powered to detect a realistic effect before treating a null as meaningful.

Key takeaways

• A confidence interval shows the range of plausible true effects; if it includes zero, the result is not significant.
• Statistical significance and practical magnitude are different; read both.
• Underpowered null results are ambiguous, not proof that a program failed.