In this city we are allowed to know the answer: 68% of bus riders are satisfied. Now run a survey and see what it reports. Choosing people at random is only half the job — the other half is who among them chooses to answer. Turn that dial and watch the estimate walk away from a truth that never moved.
one survey’s estimatethe truth, 68%what this method is aimed at
The only thing that sets the bias
Horizontal axis: how many times more likely a dissatisfied rider is to answer than a satisfied one. At 1× — and only there — the method is aimed at the truth.
68% of riders are satisfiednow find out what your survey will say
800
20%
40%
1.0%
9.0%
Your estimate—
Off the truth by—
Built-in bias—
Respondents—
Response rate—
Margin of error (95%)—
How many people answered tells you about the survey’s noise. How they were chosen — and who among them chose to reply — tells you about its aim. Only one of those two problems is cured by collecting more data.
A margin of error cannot see bias. The ± is built from the respondents alone, as though the method were pointed at the truth. Run the voluntary-response poll: thousands of replies, a margin of error under a single point, and an answer roughly 50 points wrong — with nothing anywhere in the arithmetic to warn you.