Some questions are not about how much, but about how many out of how many. Did more people recover on this treatment than on that one? The data are just counts — two numerators over two denominators — and the quantity you care about is the gap between the two shares, p̂A − p̂B. Like every statistic, that gap is not a fixed fact about the world; it is a number that would come out differently if you ran the trial again. Below you set the two true recovery rates, then run the trial for real — every patient is an actual coin flip weighted by the truth you chose. Watch how far a single trial can land from the answer. Then meet the wrinkle that makes proportions different from means: the test and the confidence interval do not use the same standard error, and there is a good reason for that.
One trial’s worth of patients
Each square is a patient (or, in a large trial, one percent of the arm); filled squares recovered. Nothing here is drawn from a formula — every square was decided by an independent weighted coin flip at the true rate you set on the right. Underneath, on a shared axis, is each arm’s observed share with its own 95% interval, and a dashed tick at the truth you are trying to recover. Press Run a new trial repeatedly and watch how much both shares move while the truth beneath them never budges.
The gap, judged against “no difference”
The curve is what p̂A − p̂B would look like across endless repeats if both arms shared one common recovery rate. That “if” is doing real work: because the null says the two rates are the same, the best estimate of that shared rate is every success over every patient, and the width of this curve is built from that pooled figure. The dark marker is the gap you actually got, and the shaded tails — every outcome at least that far from zero, in either direction — are the p‑value. The bar underneath is the confidence interval, drawn on the same scale but from a different standard error, because an interval assumes nothing about the null being true.
if both rates were equalas far out as observed, or fartherthe gap you got
Two standard errors, two different jobs
Every proportion carries its own uncertainty, √(p̂(1−p̂)/n), and the two combine the way perpendicular sides of a triangle do — square, add, take the root. That hypotenuse is the unpooled standard error, and it is what builds the confidence interval. The test uses a fourth number, the pooled standard error, computed as though both arms drew from one common rate. Watch the two bars: they run close together, they coincide exactly in the one case where the two observed shares are equal, and otherwise the pooled one sits a little higher. Knowing which belongs where is most of what this section asks of you.
Run the whole trial a thousand times
One trial settles nothing. So repeat it: each run below enrols fresh patients in both arms, flips every one of them, and records only the gap between the two recovered shares. The bars are those simulated gaps. The dark line is the normal approximation this whole section leans on — centred on the true gap pA − pB, with spread √(pAqA/nA + pBqB/nB). Nothing was fitted to the bars. When the curve and the bars agree, the approximation is earning its keep; when they part company, it is not.
simulated trialsthe normal approximationzero — no difference
Trials run0
Mean of the gaps—
SD of the gaps—
Share calling a difference—
Intervals that caught the truth—
Two treatment armsset the truth, then run the trial
The truth you are sampling from
In a real trial none of these four numbers is knowable — that is the entire problem. Here you set them, so you can grade the machinery against an answer you already have.
Arm A — the new treatment
0.55
200
Arm B — the standard treatment
0.40
200
Jump to a situation worth seeing
This trial, computed live
Arm A share p̂A—
Arm B share p̂B—
Observed gap—
True gap—you set this
SE of p̂A—√(p̂Aq̂A/nA)
SE of p̂B—√(p̂Bq̂B/nB)
SE for the interval—unpooled
SE for the test—pooled
z statistic—gap ÷ pooled SE
p‑value two‑sided—
95% confidence interval for pA − pB—
The same gap, three different headlines
A difference in proportions can be reported three ways, and they are not interchangeable. The risk difference is the gap in percentage points. The relative risk is the ratio of the two shares. The number needed to treat is how many patients you would have to give A instead of B to gain one extra recovery. The curve below holds your observed gap fixed and slides only the baseline rate underneath it — the absolute benefit never changes, yet the ratio swings wildly.
Risk difference—percentage points
Relative risk—p̂A ÷ p̂B
Number needed to treat—1 ÷ risk difference
Is the normal approximation safe here?
Everything above replaces a lumpy distribution of counts with a smooth normal curve. That swap is legitimate only when each of these four counts is comfortably large — at least 10 is the usual working rule, and some texts will accept 5. It is the counts that matter, not the sample size: a trial of five thousand patients chasing an outcome that happens twice still has nothing to smooth.
A recovered—
A did not—
B recovered—
B did not—
Two further conditions are not visible in any count. The patients must be a random sample or a randomised allocation, and the two arms must be independent of each other — different people, not the same people measured twice. Paired data (the same patients before and after) violate that second condition, and the methods on this page will misreport the uncertainty if you use them anyway. And when the counts are small, the interval drawn here — the plain p̂ ± z* SE form — stops hitting its advertised 95% reliably. The distribution underneath is a coarse ladder of a few possible values, and a recipe aimed at a smooth target cannot land on it: true coverage comes loose, usually falling short and occasionally overshooting. Drag both arms down to ten patients and run several thousand trials — one thousand is too few to see it through the noise — and watch the coverage figure settle somewhere other than the 95% on the label.
Say it right
Nine things people say after comparing two proportions. Four of them are correct. Tap each one.
What to carry out of this section
Pool for the test, do not pool for the interval. This is the one rule that is specific to proportions, and it follows from what each procedure assumes. A test starts by granting the null hypothesis that both arms share a single rate; if that rate is one number, the honest estimate of it is all the successes over all the patients, and the standard error is built from that pooled figure. A confidence interval grants nothing — it is trying to locate the gap, not to test a claim about it — so each arm keeps its own p̂ and the two variances simply add. The two standard errors sit side by side in the triangle panel: equal in the knife‑edge case where the two observed shares match exactly, and otherwise close but distinct, with the pooled one slightly the larger whenever the arms are the same size.
Variances add; standard errors do not. As with two means, Var(p̂A − p̂B) = Var(p̂A) + Var(p̂B) for independent arms — a plus sign despite the minus sign in the statistic. Independent noise cannot cancel; it can only accumulate. Adding the two standard errors instead gives the bottom bar in the triangle panel, which is always too long, and every interval built from it would be too wide.
A proportion carries its own spread. With measurements you estimate the mean and the standard deviation separately. With a proportion you do not get that choice: p̂ fixes the variability at p̂(1−p̂)/n. That product peaks at p̂ = 0.5 and collapses toward zero at either extreme, which is why rare outcomes are easier to pin down in relative terms and much harder in absolute ones, and why sample size requirements balloon near 50%.
What a small p‑value is not. It is not the probability that the two rates are equal, and it is not the probability that you are wrong. It is computed assuming they are equal, and it reports how often a gap at least this extreme would appear under that assumption. Inverting it — treating it as the probability of the assumption itself — is the most common error in applied statistics, and no sample size protects you from it.
What 95% confidence is not. The interval either contains the true gap or it does not; once the arithmetic is done there is nothing left to be random about. The 95% describes the procedure: across endless repeats, that recipe brackets the truth 95% of the time. The simulation panel measures exactly that, one interval per trial, and reports the running score — including the honest fact that with small counts this particular recipe stops delivering the number printed on its label.
Relative risk without a baseline is not information. Load the Rare outcome preset. The rates you set are 3% and 1%: a relative risk of three, which sounds enormous, and an absolute gap of two percentage points, which means about fifty patients treated per extra recovery. Both numbers describe the same truth. Your particular trial will report something near them rather than exactly them — with events this scarce, a handful of patients either way moves the ratio a long distance and the absolute gap barely at all, which is itself the lesson. Only one of them tells you what happens to a patient. Whenever a ratio is quoted without the baseline it was computed from, the baseline is the missing number, not an optional detail.
A gap between arms is not automatically caused by the arms. Every number on this page would be identical whether patients were randomised to A and B or simply sorted themselves into the two groups. The statistics measure the size and precision of a gap; they are silent about what produced it. Only random allocation licenses a causal reading, and no value of z upgrades an observational comparison into a causal one.