Selection Effects

The comparison-group module explained why voluntary participation biases results at the start. Selection does not stop at enrollment. It operates continuously, through who joins, who stays, and who leaves, and each channel can manufacture a result on its own.

Three ways selection strikes

• Joining (self-selection). Providers who opt into a program differ from those who do not, often healthier panels, better management. Favorable results may reflect who joined, not what the program did. This is the confounding from Module 1.
• Leaving (attrition). Participants who do poorly are more likely to quit. If the strugglers drop out, the survivors’ average improves automatically, with no one actually getting better. An evaluation that measures only those who stayed is measuring a group reshaped by its own exits.
• Cream-skimming and dumping. When payment depends on the population, organizations have an incentive to attract profitable patients and shed unprofitable ones. A “good” result can reflect a favorable patient mix engineered after the fact.

Worth remembering: ask not just who joined, but who left and why. A program whose sickest patients or weakest providers quietly exited will look better every year it sheds them, and that improvement is pure selection.

The Pioneer example

The Pioneer ACO model from the case-studies module shows selection in the open. Of the original 32 organizations, more than a third left within two years, most because they faced or feared losses. Any evaluation of “how Pioneer ACOs performed” has to grapple with the fact that the worst-fitting participants removed themselves. The organizations that stayed were not a random remnant; they were the ones for whom the model worked, which flatters the surviving average.

How careful studies handle it

• Intent-to-treat analysis counts everyone who started, including dropouts, so attrition cannot inflate the result.
• Reporting the exits (who left and why) lets readers judge the bias themselves.
• Comparing entry and exit characteristics shows whether the population was stable or reshaped.

When a study reports only on continuing participants and stays silent about who left, treat its results as an upper bound at best.

Key takeaways

• Selection operates through joining, leaving, and patient mix, and each can create a result with no real effect.
• Attrition is the subtle one: when the weakest members exit, the average improves on its own.
• Look for intent-to-treat analysis and honest reporting of who left; their absence signals selection bias.