Chi-Square & ANOVA

Every test you have met so far has compared one mean, or two. Real questions are rarely that tidy. You want to know whether a whole set of categories falls out the way somebody claimed, whether two categorical variables travel together, or whether three, four, or five groups have the same average. This chapter gives you three tools for exactly those jobs — the chi-square goodness-of-fit test, the chi-square test of independence, and one-way analysis of variance — and then it tells you the truth that most courses rush past: what these tests require before you may trust them, and what to do the moment one of them comes back significant. The arithmetic is mostly subtraction, squaring, and division. If you can do that, you can do this.

Section 1Chi-square goodness of fit

The question this test answers

You have one categorical variable with k categories, and somebody has made a claim about how the population splits across those categories. A manufacturer says its bags of candy are 30% red, 30% brown, and 40% yellow. A textbook says a fair die lands on each face one-sixth of the time. A subscription service says its customers split 25/40/35 across three tiers. You collect a sample and count. The counts never match the claim exactly — sampling sees to that. The goodness-of-fit test asks a single question: is the mismatch bigger than sampling noise can comfortably explain?

The hypotheses are stated in words as much as symbols:

• H₀: the population proportions are exactly as claimed (p₁ = 0.25, p₂ = 0.40, p₃ = 0.35, say).
• H_a: at least one proportion differs from its claimed value.

Notice that the alternative is vague on purpose. It says something is off; it does not say which category or in which direction. That vagueness is the price of testing all the categories at once, and it comes back to bite us in Section 5 in the ANOVA setting.

Observed, expected, and the statistic

The observed counts O are what you actually tallied. The expected counts E are what the null hypothesis predicts on average for a sample of your size: E_i = n × p_i. Expected counts are frequently not whole numbers — 200 × 0.35 = 70 is tidy, but 137 × 0.35 = 47.95 is not, and you must not round it. Rounding expected counts is one of the most common ways a chi-square gets quietly wrecked by hand.

The statistic adds up one contribution per category:

χ² = Σ (O − E)² / E

Read that formula slowly, because it is the whole chapter in miniature. The numerator, (O − E)², measures how far a category strayed from prediction, squared so that overshoots and undershoots both count as evidence. Dividing by E puts that miss on a fair scale: being 10 counts off when you expected 20 is a scandal; being 10 counts off when you expected 2,000 is nothing. Because every term is a square divided by a positive number, χ² is never negative, and a value near zero means the data hug the claim. Big values, and only big values, are evidence against H₀ — which is why the chi-square test is always right-tailed.

The degrees of freedom are df = k − 1, one less than the number of categories. The intuition: once you know the total and all but one category count, the last one is forced, so only k − 1 counts are free to vary. (If you had to estimate m parameters from the data to build the expected counts — fitting a Poisson model, for instance — you subtract those too: df = k − 1 − m.)

A worked example, start to finish

A meal-plan service claims its subscribers split across three tiers as Bronze 25%, Silver 40%, Gold 35%. A random sample of n = 200 subscribers gives Bronze 40, Silver 95, Gold 65. Test at α = 0.05.

Step 0 — check the conditions first. The four requirements are laid out in Section 4, and they are met here: the data are raw counts, not percentages; the sample is random; each subscriber is counted once and lands in exactly one tier, so the observations are independent; and every expected count computed in Step 1 is at least 5 (the smallest is 50). Do this before you look at any p-value, not after.

Step 1 — expected counts. E_Bronze = 200(0.25) = 50; E_Silver = 200(0.40) = 80; E_Gold = 200(0.35) = 70. They sum to 200, which is a useful arithmetic check — expected counts always total n.

Step 2 — the statistic. 2.0000 + 2.8125 + 0.3571 = χ² = 5.17.

Step 3 — degrees of freedom and critical value. k = 3 categories, so df = 3 − 1 = 2. From the table below, the critical value at α = 0.05 with df = 2 is 5.991.

Step 4 — decide. 5.17 is less than 5.991, so we fail to reject H₀. The p-value is about 0.075, which is larger than 0.05 and confirms the same decision.

Step 5 — say what that means, carefully. "At the 5% level, these data do not provide sufficient evidence that the true tier proportions differ from the claimed 25/40/35 split." That sentence is doing something specific. It does not say the claim is true. We fail to reject H₀; we never accept it. Those are different verdicts — "not guilty" is not "innocent" — and a sample of 200 simply may not have had the power to detect a mismatch of the size that is really there. Absence of evidence is not evidence of absence. And the p-value of 0.075 is not a 7.5% chance that the provider is right. It is this: if the claimed split were exactly correct, samples of 200 would produce a mismatch this extreme or worse about 7.5% of the time. The probability lives in the data-generating process, not in the hypothesis.

Section 2Chi-square test of independence

Two variables, one table

Now you have two categorical variables measured on the same people, and you cross them into a contingency table (also called a two-way table or crosstab) with r rows and c columns. Does exercise habit go with sleep quality? Does region go with brand preference? Does treatment group go with outcome category? The test of independence asks whether knowing a person's row tells you anything about their column.

• H₀: the two variables are independent in the population — there is no association.
• H_a: the two variables are associated — they are not independent.

The statistic is the same formula as Section 1, cell by cell, and the test is still right-tailed. Only two things change: how you get the expected counts, and how you count degrees of freedom.

Expected counts from the margins

Under independence, the probability of landing in a particular row and a particular column is the product of the two separate probabilities. Translate that into counts and you get the working formula every cell obeys:

E = (row total × column total) / grand total

Why it works: the row total divided by n estimates the probability of that row; multiply by the column total and you get how many of that column's members should fall in that row if the row split were the same everywhere. That last phrase is the practical meaning of independence — every row has the same column profile.

Degrees of freedom are df = (r − 1)(c − 1). Same logic as before: with the margins fixed, once you fill in an (r − 1) by (c − 1) block of the table, subtraction forces every remaining cell.

A worked example, start to finish

A researcher surveys n = 300 adults, recording whether they exercise regularly and how they rate their sleep. Test for association at α = 0.05.

Step 1 — expected counts. Top-left: (150 × 120)/300 = 60.0. Top-middle: (150 × 105)/300 = 52.5. Top-right: (150 × 75)/300 = 37.5. Because the two row totals happen to be equal here, the bottom row repeats those values. Every expected count is at least 5, so the test is safe to run.

Step 2 — the statistic. Adding the last column: χ² = 16.07.

Step 3 — degrees of freedom. r = 2 rows, c = 3 columns, so df = (2 − 1)(3 − 1) = 2. Critical value at α = 0.05 is again 5.991.

Step 4 — decide and interpret. 16.07 far exceeds 5.991, and the p-value is roughly 0.0003. We reject H₀: there is strong evidence of an association between regular exercise and self-rated sleep quality in this population.

Step 5 — describe the association. The test says "associated" and stops there, so you have to look at the cells yourself. The two largest contributions are the two "Poor" cells at 4.1667 each — Exercise-and-Poor (observed 25, expected 37.5) and No-exercise-and-Poor (observed 50, expected 37.5) — followed by the two "Good" cells at 3.7500 each. Because the row totals happen to be equal here, the contributions come in mirrored pairs, so it is the Good and Poor columns together that drive the result: exercisers landed in "Good" more often and "Poor" less often than independence predicts, and non-exercisers did the reverse. A tidier way to say the same thing is the standardized residual, (O − E)/√E; for Exercise-and-Good that is 15/√60 = +1.94, and for Exercise-and-Poor it is −12.5/√37.5 = −2.04, with the no-exercise row reversing both signs. Residuals beyond roughly ±2 mark the cells that drove the result — here only the two Poor cells clear that bar, with the two Good cells just short of it.

Step 6 — measure how big it is. With 300 people, even a modest association gets a tiny p-value, so report an effect size. Cramér's V = √[χ² / (n × min(r−1, c−1))] = √[16.071 / (300 × 1)] = √0.05357 = 0.23. On a 0-to-1 scale, that is a real but moderate association — statistically unmistakable, practically modest.

Section 3One-way ANOVA

Why not just run a lot of t tests?

Suppose you have three training methods and you want to know whether their mean test scores differ. The obvious move is three t tests: A vs B, A vs C, B vs C. The obvious move is wrong, and it is worth seeing exactly why.

Each test at α = 0.05 has a 5% chance of a false alarm. If the three tests were independent, the chance that at least one raises a false alarm would be 1 − (0.95)³ = 1 − 0.857 = 0.143. With five groups there are ten pairwise tests, and the figure climbs to 1 − (0.95)¹⁰ = 0.401. Be honest about that calculation: pairwise tests on the same groups are not independent, because every pair reuses a sample the other pairs also used. So treat 14% and 40% as illustrations of the size of the problem rather than exact rates. The direction is what matters, and it is not in doubt — you set out to run a 5% risk and quietly took on a far larger one. This inflated risk is the family-wise error rate, and it is the reason ANOVA exists: one test, one α, one answer.

• H₀: μ₁ = μ₂ = μ₃ = … = μ_k (all population means are equal).
• H_a: at least one mean differs from at least one other. (Not "all means differ" — a very common misstatement.)

The idea: two ways to measure spread

The name is honest — you analyze variance in order to learn about means. Here is the trick. If all the group means were truly equal, then two completely different calculations would both estimate the same population variance σ²:

• Within-group variability (MSW): pool the scatter inside each group. Nothing about group differences touches this number — it is pure background noise, and it estimates σ² whether H₀ is true or not.
• Between-group variability (MSB): measure how far the group means sit from the grand mean. If H₀ is true, the means differ only by sampling noise and this also estimates σ². If H₀ is false, real mean differences inflate it.

So the ratio

F = MSB / MSW = (signal + noise) / noise

should sit near 1 when the means are equal and grow large when they are not. That is why ANOVA, like chi-square, is a right-tailed test — only big F values are evidence against H₀. And because both mean squares are sums of squares divided by positive degrees of freedom, neither can be negative, so F is never negative either.

The building blocks, with N = total observations, k = number of groups, n_j and x̄_j the size and mean of group j, and x̄ the grand mean:

• SS_between = Σ n_j(x̄_j − x̄)², with df = k − 1
• SS_within = ΣΣ(x − x̄_j)², with df = N − k
• SS_total = SS_between + SS_within, with df = N − 1
• MS = SS / df in each row, and F = MSB/MSW with (k − 1, N − k) degrees of freedom

A worked example, start to finish

Three training methods, five participants each (N = 15, k = 3), test scores. Conditions first, as always: the groups are independent, each is roughly symmetric with no outliers, the response is quantitative, and the equal-variance condition is checked explicitly with these very numbers in Section 4. ANOVA is the right tool here.

Step 1 — grand mean. (360 + 385 + 410)/15 = 1155/15 = 77.

Step 2 — between-group sum of squares. SS_between = 5(72 − 77)² + 5(77 − 77)² + 5(82 − 77)² = 5(25) + 5(0) + 5(25) = 125 + 0 + 125 = 250.

Step 3 — within-group sum of squares. Add the last column of the table: 34 + 58 + 46 = 138.

Step 4 — check the partition. SS_total should equal 250 + 138 = 388. Computing it directly from all fifteen deviations about 77 gives 159 + 58 + 171 = 388. The identity holds, which is a strong sign the arithmetic is right.

Step 5 — mean squares and F. MSB = 250/2 = 125.00. MSW = 138/12 = 11.50. F = 125.00/11.50 = 10.87.

Step 6 — decide. The critical value for F(2, 12) at α = 0.05 is 3.89. Our 10.87 sails past it, p ≈ 0.0020, so we reject H₀.

Step 7 — interpret precisely. "There is strong evidence that at least one of the three method means differs from the others." That is all the F test earns you. It does not say A differs from B, and it does not say every pair differs. Section 5 is where you find out which pairs are really separated — and the answer will surprise you.

Step 8 — report an effect size. Eta squared, η² = SS_between/SS_total = 250/388 = 0.64. About 64% of the variation in test scores is explained by which method a participant received — a large effect, and a far more useful number to a decision-maker than "p < 0.05."

Section 4Assumptions

What chi-square requires

A p-value is a promise, and the promise is conditional. Every chi-square p-value assumes four things, and software will happily print a number even when all four are violated.

1. The data are counts. Chi-square consumes raw frequencies — how many people, how many items. Feeding it percentages, proportions, or group means is the single most frequent misuse. If your table holds "42%" rather than "42 people," convert back to counts first, and if you cannot, you do not have the data you need.

2. Observations are independent, and each falls in exactly one cell. One row of data per person or item. If the same person is counted twice — before and after an intervention, say — the observations are paired and chi-square is the wrong test; use McNemar's test for a paired 2 × 2 table.

3. Expected counts are large enough. The chi-square curve is a continuous approximation to a discrete count problem, and the approximation frays when expected counts are tiny. The common working rule: all expected counts ≥ 5. Cochran's slightly more forgiving version allows up to 20% of cells to have E between 1 and 5, provided none falls below 1. Note that this is a condition on expected, not observed, counts — an observed zero is perfectly legal.

4. The sample is random. Every inferential claim depends on the sample representing the population. No formula repairs a convenience sample.

When rule 3 fails, you have real options: collapse sparse categories into a sensible larger one (two adjacent, rarely used response levels become a single "other" category), collect more data, or switch to Fisher's exact test, which computes the probability directly and needs no large-sample approximation at all.

What ANOVA requires

1. Independence, both within and between groups. This is the assumption ANOVA is least willing to forgive, and no amount of data fixes a violation. If the same subjects appear in every group, you need repeated-measures ANOVA instead.

2. Approximate normality of the response within each group (equivalently, of the residuals). ANOVA is genuinely robust here: with roughly equal group sizes and a sample of decent size, moderate skew barely dents the F test, and the Central Limit Theorem is on your side. The condition matters most when groups are small (under about 15) or the data are strongly skewed or heavily outlier-ridden. Check it with a histogram or normal quantile plot per group, not with a formal normality test on tiny samples.

3. Homogeneity of variance — the population spreads are equal across groups. MSW pools all the groups' scatter into one number, and that pooling only makes sense if there is one common σ² to pool. Two quick checks: the largest sample standard deviation should be less than about twice the smallest, or equivalently the largest variance less than about four times the smallest. Levene's test is the formal version.

4. The response is quantitative and measured on an interval or ratio scale. You cannot average a category.

Let us actually run check 3 on the Section 3 data. The within-group sums of squares were 34, 58, and 46, each with df = 4:

Largest s divided by smallest s = 3.81/2.92 = 1.31, comfortably under 2. Largest variance over smallest = 14.50/8.50 = 1.71, comfortably under 4. The equal-variance condition is satisfied, so the pooled MSW of 11.50 was a legitimate thing to compute.

When a condition fails

A failed assumption is not the end of the analysis. It is a signpost to a different test.

Section 5After a significant ANOVA

The F test leaves you with a question

We rejected H₀ in Section 3 with F = 10.87. What we know is that the three method means are not all equal. What we do not know is which ones differ. That gap is exactly what post-hoc ("after this") comparisons fill: they go back through the pairs, one at a time, while keeping the total false-alarm risk across the whole family of comparisons at α.

And this is where you cannot simply run ordinary t tests, for the very reason ANOVA existed in the first place. Three unadjusted pairwise tests carry a family-wise error rate in the neighborhood of 14%. Post-hoc procedures are pairwise tests with the inflation engineered out.

Two workhorses: Bonferroni and Tukey

The Bonferroni correction is the simplest honest fix in all of statistics: if you plan m comparisons, test each one at α/m. With three groups, m = 3 and each comparison is judged at 0.05/3 = 0.0167. Equivalently, multiply each raw p-value by 3 and compare to 0.05. It works for any set of comparisons, it takes ten seconds to explain, and it is somewhat conservative — it protects against false alarms so firmly that it can miss real differences, especially when m is large.

Tukey's HSD (Honestly Significant Difference) is purpose-built for the specific job of comparing all pairs after an ANOVA, and for that job it is more powerful than Bonferroni. Instead of a t distribution it uses the studentized range distribution q, which already accounts for the fact that you are looking at the largest gaps among k means. For equal group sizes the rule is beautifully simple: any two means further apart than the HSD value are significantly different.

HSD = q_{α, k, N−k} × √(MSW / n)

Working it out on our data

From the ANOVA table: MSW = 11.50, N − k = 12, k = 3, n = 5 per group. The studentized range value q for α = 0.05, k = 3, df = 12 is 3.77.

Step 1 — the standard error piece. √(MSW/n) = √(11.50/5) = √2.30 = 1.517.

Step 2 — the HSD threshold. HSD = 3.77 × 1.517 = 5.72. Any two sample means more than 5.72 points apart are significantly different; any pair closer than that is not.

Step 3 — compare every pair. The means were A = 72, B = 77, C = 82.

Read that result carefully, because it is the whole point of the section. The overall F was highly significant, yet only one of the three pairs separates. Method C beats method A; neither can be distinguished from the middle method B. An analyst who reported "the ANOVA was significant, so all three methods differ" would have gotten the science wrong while getting the arithmetic right.

Step 4 — give intervals, not just verdicts. Tukey also produces confidence intervals: difference ± HSD. For A vs C that is −10 ± 5.72, or (−15.72, −4.28). Because the interval excludes zero, the difference is significant — and the interval says far more than the p-value does, namely that the plausible advantage of C over A runs from about 4 to about 16 points. Say what such an interval means with care: the procedure that produced it captures the true mean difference in 95% of all possible samples. It is not a 95% chance that this particular interval, already computed and sitting on the page, contains the parameter. The parameter is fixed; the interval is what varies from sample to sample.

Step 5 — cross-check with Bonferroni. The standard error of a difference is √[MSW(1/n + 1/n)] = √[11.50(0.4)] = √4.60 = 2.145. For A vs C, t = 10/2.145 = 4.66; for A vs B, t = 5/2.145 = 2.33. The Bonferroni critical value is the t for α = 0.0167 two-tailed with df = 12, about 2.78. So A vs C (4.66) clears it and A vs B (2.33) does not — the same conclusion Tukey reached, which is reassuring.

Choosing a method, and when not to bother

Two habits keep this honest. First, if the overall F is not significant, do not go fishing through the pairs hoping one turns up — that is precisely the error-inflation the F test protected you from. (Planned contrasts specified before seeing the data are the legitimate exception.) Second, always show the picture. Side-by-side boxplots of the three groups convey the overlap between A and B, and the clean separation between A and C, more immediately than any table of p-values.

ReferenceWords worth knowing

A quick reference for the terms in this chapter — skim it now, or circle back whenever a word trips you up.

Alpha (α)

The significance level: the risk of a false alarm you agree to accept before collecting data, most often 0.05.

Analysis of variance (ANOVA)

A procedure that compares three or more population means by comparing two estimates of variance, between-group and within-group.

Bonferroni correction

Testing each of m planned comparisons at α/m so the family-wise error rate stays at α. Simple and conservative.

Chi-square statistic (χ²)

Σ(O − E)²/E, a non-negative measure of total mismatch between observed and expected counts.

Contingency table

A two-way table of counts crossing two categorical variables, with row, column, and grand totals in the margins.

Cramér's V

An effect size for a contingency table, √[χ²/(n × min(r−1, c−1))], running from 0 (no association) to 1.

Degrees of freedom

The number of values free to vary once the constraints are imposed; k − 1 for goodness of fit, (r−1)(c−1) for independence, and (k−1, N−k) for ANOVA.

Eta squared (η²)

SS_between/SS_total: the proportion of variation in the response explained by group membership.

Expected count (E)

The count a cell would show on average if the null hypothesis were true. Never round it.

F statistic

The ratio MSB/MSW. Near 1 when group means are equal, large when they are not; never negative, and always tested in the upper tail.

Family-wise error rate

The probability of at least one false alarm across a whole family of tests. It inflates rapidly as the number of comparisons grows.

Goodness-of-fit test

A chi-square test of whether one categorical variable's observed counts match a claimed distribution.

Homogeneity of variance

The ANOVA condition that all groups share the same population variance; checked with s_max/s_min < 2 or Levene's test.

Kruskal–Wallis test

A rank-based alternative to one-way ANOVA used when the normality condition fails badly.

Mean square (MS)

A sum of squares divided by its degrees of freedom — that is, a variance estimate. MSB and MSW are the two that build F.

Observed count (O)

The actual tally in a category or cell of your sample.

p-value

The probability of getting data at least as extreme as yours if the null hypothesis were true. It is not the probability that the null hypothesis is true.

Post-hoc test

A follow-up comparison run after a significant ANOVA to identify which specific groups differ, with the family-wise error rate controlled.

Standardized residual

(O − E)/√E for one cell; values beyond roughly ±2 flag the cells driving a significant chi-square.

Sum of squares (SS)

A total of squared deviations. In ANOVA, SS_total splits exactly into SS_between + SS_within.

Test of independence

A chi-square test of whether two categorical variables measured on the same subjects are associated.

Tukey's HSD

Honestly Significant Difference: q × √(MSW/n), the standard post-hoc method for comparing all pairs of means after ANOVA.