37. Chi-squared tests

The t-test compares means of numeric data. When the data is counts in categories — how many people chose each option, how many defects of each type — the \chi^2 family takes over.

Two tests, one statistic.

The statistic

Both tests compare observed counts against what a hypothesis predicts:

\chi^2 = \sum_{\text{cells}} \frac{(O - E)^2}{E}

where O is observed and E is expected under H_0.

Every part of that formula earns its place:

• (O - E) — the discrepancy. Squared, so over- and under-shoots both count.
• Divided by E — this is the crucial bit. A discrepancy of 10 is enormous if you expected 5 and trivial if you expected 10,000. Dividing by E standardises: each term is roughly a squared z-score, since a count with mean E has variance about E.

Summing squared standard Normals gives a chi-squared distribution (§1) — which is where the name and the reference distribution come from.

Large \chi^2 means the observed counts are far from what H_0 predicts. The test is always one-sided in the upper tail, because only large discrepancies are evidence against H_0. (A suspiciously small \chi^2 means the data fits too well — historically a sign of fabricated data, but not what the test is designed to detect.)

Test 1: goodness of fit

Question: does the data follow a claimed distribution?

H_0: the proportions are p_1, \dots, p_k.

E_i = n p_i, \qquad df = k - 1 - (\text{number of parameters estimated from the data})

Why k - 1? Because the counts must sum to n, so once you know k-1 of them the last is determined. One constraint, one degree of freedom lost.

And every parameter you estimate from the data costs another. Testing whether data is Poisson without knowing \lambda means estimating it, so df = k - 1 - 1.

Test 2: independence (contingency tables)

Question: are two categorical variables related?

H_0: the row and column variables are independent.

Under independence, $P(\text{row } i \text{ and col } j) = P(\text{row } i) \times P(\text{col } j)$, and estimating those probabilities from the margins gives

E_{ij} = \frac{(\text{row total}_i)(\text{column total}_j)}{n}

df = (r - 1)(c - 1)

The degrees of freedom count the free cells: fix r-1 rows and c-1 columns and every remaining cell is determined by the margins.

The assumptions

The \chi^2 distribution is an approximation, valid when counts are large enough. The usual rules:

• All expected counts E_i \ge 5 — or at least 80% of cells \ge 5 and none below 1.
• Observations independent; each observation in exactly one cell.
• Use raw counts, never percentages. Feeding in percentages silently pretends n = 100, which fabricates or destroys evidence.

When expected counts are too small, use Fisher's exact test (for 2×2 tables) or pool categories.

Worked example 1: goodness of fit

A die is rolled 120 times: 15, 22, 18, 25, 20, 20. Is it fair?

Under H_0 each face has probability 1/6, so E_i = 120/6 = 20 for all six.

\chi^2 = \frac{(15-20)^2}{20} + \frac{(22-20)^2}{20} + \frac{(18-20)^2}{20} + \frac{(25-20)^2}{20} + \frac{(20-20)^2}{20} + \frac{(20-20)^2}{20}

= \frac{25 + 4 + 4 + 25 + 0 + 0}{20} = \frac{58}{20} = 2.9

df = 6 - 1 = 5, and the critical value is \chi^2_{5,\,0.05} = 11.07.

Since 2.9 < 11.07, fail to reject. The p-value is about 0.715 — this data is entirely consistent with a fair die.

Worked example 2: independence

Is treatment related to recovery?

	Recovered	Not recovered	Total
Treatment	45	15	60
Placebo	30	30	60
Total	75	45	120

Expected counts under independence:

E_{11} = \frac{60 \times 75}{120} = 37.5, \qquad E_{12} = \frac{60 \times 45}{120} = 22.5

and the same for row 2, by symmetry of the margins.

\chi^2 = \frac{(45-37.5)^2}{37.5} + \frac{(15-22.5)^2}{22.5} + \frac{(30-37.5)^2}{37.5} + \frac{(30-22.5)^2}{22.5}

= 1.5 + 2.5 + 1.5 + 2.5 = 8.0

df = (2-1)(2-1) = 1, critical value \chi^2_{1,\,0.05} = 3.84.

8.0 > 3.84, so reject — treatment and recovery are associated (p \approx 0.0047).

But note what the test does not tell you. It says the variables are associated; it doesn't give the direction or size. For that, report the proportions: 75% recovered on treatment against 50% on placebo, a difference of 25 percentage points. The \chi^2 statistic is a detector, not a description.

Doing it in Python

Both tests, with SciPy:

import numpy as np
from scipy.stats import chisquare, chi2_contingency, chi2

# Goodness of fit: is the die fair?
observed = np.array([15, 22, 18, 25, 20, 20])
result = chisquare(observed)                  # expects uniform by default
print(f"goodness of fit: chi2 = {result.statistic:.4f}  p = {result.pvalue:.4f}")
print(f"  critical value at 5%: {chi2.ppf(0.95, df=5):.4f}")

# Independence: treatment vs recovery
table = np.array([[45, 15],
                  [30, 30]])
stat, p, dof, expected = chi2_contingency(table, correction=False)
print(f"\nindependence: chi2 = {stat:.4f}  df = {dof}  p = {p:.6f}")
print("expected counts under independence:\n", expected)

Building the statistic by hand, so nothing is a black box:

import numpy as np
from scipy.stats import chi2

table = np.array([[45, 15],
                  [30, 30]], dtype=float)

n = table.sum()
row_totals = table.sum(axis=1, keepdims=True)
col_totals = table.sum(axis=0, keepdims=True)
expected = row_totals @ col_totals / n

print("expected:\n", expected)
print("\ncontribution of each cell to chi2:")
contrib = (table - expected) ** 2 / expected
print(contrib.round(4))

stat = contrib.sum()
dof = (table.shape[0] - 1) * (table.shape[1] - 1)
print(f"\nchi2 = {stat:.4f}, df = {dof}, p = {1 - chi2.cdf(stat, dof):.6f}")

# And the effect size the test itself doesn't give you
p_treat = table[0, 0] / table[0].sum()
p_placebo = table[1, 0] / table[1].sum()
print(f"\nrecovery: {p_treat:.1%} on treatment vs {p_placebo:.1%} on placebo")
print(f"difference: {p_treat - p_placebo:.1%} points")

When the approximation breaks — small expected counts:

import numpy as np
from scipy.stats import chi2_contingency, fisher_exact

small = np.array([[8, 2],
                  [1, 5]])
stat, p_chi, dof, expected = chi2_contingency(small, correction=False)
_, p_fisher = fisher_exact(small)

print("table:\n", small)
print("expected counts:\n", expected.round(3))
print(f"\nsmallest expected count: {expected.min():.3f}  (rule of thumb wants >= 5)")
print(f"chi2 p-value  : {p_chi:.4f}")
print(f"Fisher exact p: {p_fisher:.4f}   <- trust this one here")

And a demonstration that the reference distribution really is \chi^2:

import numpy as np
from scipy.stats import chi2

rng = np.random.default_rng(0)
n, k, trials = 200, 6, 50_000

# Sample from a genuinely fair die, so H0 is TRUE
counts = rng.multinomial(n, [1/k] * k, size=trials)
expected = n / k
stats_ = ((counts - expected) ** 2 / expected).sum(axis=1)

print(f"simulated mean {stats_.mean():.4f}   chi2_{k-1} mean {k-1}")
print(f"simulated var  {stats_.var():.4f}   chi2_{k-1} var  {2*(k-1)}")
for q in (0.9, 0.95, 0.99):
    print(f"  q={q}: simulated {np.quantile(stats_, q):.4f}   theory {chi2.ppf(q, k-1):.4f}")
print(f"\nfalse positive rate at 5%: {(stats_ > chi2.ppf(0.95, k-1)).mean():.4f}")

Your turn

1. 100 people choose among 4 options: 30, 20, 25, 25. Test whether all options are equally popular.

2. What are the degrees of freedom for a 3×4 contingency table?

3. Why can't you run a \chi^2 test on percentages?

Solutions

1. Under H_0 each option has probability 1/4, so E_i = 25 for all four.

\chi^2 = \frac{(30-25)^2 + (20-25)^2 + (25-25)^2 + (25-25)^2}{25} = \frac{25 + 25 + 0 + 0}{25} = 2.0

df = 4 - 1 = 3, and \chi^2_{3,\,0.05} = 7.81.

Since 2.0 < 7.81, fail to reject (p \approx 0.572). The data is consistent with equal popularity.

All expected counts are 25, comfortably above 5, so the approximation is fine.

2.

df = (r-1)(c-1) = (3-1)(4-1) = 6

3. Because the \chi^2 statistic depends on the sample size, and percentages destroy that information.

Suppose you observe a 60/40 split. As raw counts:

• 6 vs 4 out of 10: \chi^2 = \frac{1 + 1}{5} = 0.4, clearly not significant.
• 600 vs 400 out of 1000: \chi^2 = \frac{100 + 100}{500} = 40, overwhelmingly significant.

Same percentages, hundredfold difference in evidence. Entering "60 and 40" as if they were counts silently claims n = 100 — inventing evidence if your real sample was smaller, discarding it if larger.

The same reasoning explains why you must not enter counts that are already aggregated over repeated measures on the same subject: the test assumes each count is one independent observation.

Check yourself in code

Run both \chi^2 tests — the fair-die goodness of fit and the treatment independence test — and confirm the by-hand statistic matches SciPy.

Print exactly this:

die chi2 2.9
die p 0.7154
table chi2 8.0
table p 0.0047
by hand matches: True

Round the statistics to 4 decimal places and the p-values to 4. Use correction=False for the contingency test.

import numpy as np
from scipy.stats import chisquare, chi2_contingency

observed = np.array([15, 22, 18, 25, 20, 20])
die = chisquare(observed)
print("die chi2", round(die.statistic, 4))
print("die p", round(die.pvalue, 4))

table = np.array([[45, 15],
                  [30, 30]], dtype=float)

# Run chi2_contingency with correction=False, print the statistic and p-value,
# then recompute the statistic by hand from the margins and compare.

import numpy as np
from scipy.stats import chisquare, chi2_contingency

observed = np.array([15, 22, 18, 25, 20, 20])
die = chisquare(observed)
print("die chi2", round(die.statistic, 4))
print("die p", round(die.pvalue, 4))

table = np.array([[45, 15],
                  [30, 30]], dtype=float)
stat, p, dof, expected = chi2_contingency(table, correction=False)
print("table chi2", round(stat, 4))
print("table p", round(p, 4))

n = table.sum()
by_hand = (((table - table.sum(axis=1, keepdims=True)
             @ table.sum(axis=0, keepdims=True) / n) ** 2)
           / (table.sum(axis=1, keepdims=True)
              @ table.sum(axis=0, keepdims=True) / n)).sum()
print("by hand matches:", round(by_hand, 4) == round(stat, 4))

---

The \chi^2 statistic sums (O-E)^2/E across cells, standardising each discrepancy by how big it was expected to be. Use it for goodness of fit against a claimed distribution, or for independence in a contingency table, with degrees of freedom counting the genuinely free cells. It needs raw counts and expected values of at least about 5 — and it detects association without describing it.

Next: comparing more than two groups at once, where the F distribution appears.