t-tests · Probability & Statistics

The t-test is the workhorse of applied statistics: comparing means when the population standard deviation is unknown, which is essentially always.

There are three variants, and choosing the wrong one is the most common mistake in practice.

Why t and not z

If \sigma were known,

$Z = \frac{\bar X - \mu}{\sigma/\sqrt n} \sim N(0,1)$

exactly. Replace \sigma with the sample standard deviation s and the statistic picks up a second source of randomness — s varies from sample to sample too. The result is no longer Normal:

$T = \frac{\bar X - \mu}{s/\sqrt n} \sim t_{n-1}$

The t distribution has heavier tails, so its critical values are larger and its intervals wider. That extra width is the honest price of not knowing \sigma.

Where it comes from: T is a standard Normal divided by the square root of an independent scaled \chi^2_{n-1} — precisely the construction from §1. The n-1 degrees of freedom is the same one that appears in s^2.

As n grows, s \to \sigma and t_{n-1} \to N(0,1) — Slutsky's theorem again (§3).

1. One-sample t-test

Question: does this sample come from a population with mean \mu_0?

$H_0: \mu = \mu_0, \qquad t = \frac{\bar x - \mu_0}{s/\sqrt n}, \qquad df = n - 1$

Assumptions: independent observations; data approximately Normal or n large enough for the CLT.

2. Two-sample t-test (independent groups)

Question: do two independent groups have the same mean?

$H_0: \mu_1 = \mu_2, \qquad t = \frac{\bar x_1 - \bar x_2}{\text{SE}}$

There are two versions, differing in how they estimate the SE.

Welch's t-test (unequal variances — the default you should use):

$\text{SE} = \sqrt{\frac{s_1^2}{n_1} + \frac{s_2^2}{n_2}}$

with a fractional degrees of freedom given by the Welch–Satterthwaite formula.

Pooled (Student's) t-test (assumes \sigma_1 = \sigma_2):

$s_p^2 = \frac{(n_1-1)s_1^2 + (n_2-1)s_2^2}{n_1 + n_2 - 2}, \qquad \text{SE} = s_p\sqrt{\frac{1}{n_1} + \frac{1}{n_2}}$

with df = n_1 + n_2 - 2.

Use Welch by default. It's barely less powerful when the variances really are equal, and it stays valid when they aren't — especially with unequal group sizes, where the pooled test's error rate can be badly wrong. R's t.test defaults to Welch; SciPy's ttest_ind defaults to pooled, so you must pass equal_var=False.

And don't pre-test for equal variances to decide: the two-stage procedure distorts the Type I rate of whatever you run second.

3. Paired t-test

Question: for matched observations, is the mean difference zero?

Before/after on the same subjects, twins, left/right eye, matched pairs.

Compute the differences d_i = x_i - y_i and run a one-sample test on them:

$H_0: \mu_d = 0, \qquad t = \frac{\bar d}{s_d/\sqrt n}, \qquad df = n - 1$

This is the variant people get wrong most often. Using an independent two-sample test on paired data throws away the pairing, which is usually where most of the precision lives. If subjects differ a lot from each other but respond consistently, the between-subject variation swamps the effect in a two-sample test and cancels exactly in a paired one.

Conversely, using a paired test on unpaired data is simply invalid — there are no pairs to difference.

Worked example

A drug is tested on 12 patients, blood pressure measured before and after:

Before	145	152	138	160	149	155	142	158	147	151	144	156
After	138	145	135	150	144	147	140	149	143	145	141	148

Same patients measured twice, so this is paired.

Differences (before − after): 7, 7, 3, 10, 5, 8, 2, 9, 4, 6, 3, 8.

$\bar d = \frac{72}{12} = 6.0, \qquad s_d \approx 2.594$

$t = \frac{6.0}{2.594/\sqrt{12}} = \frac{6.0}{0.749} \approx 8.01, \qquad df = 11$

With t_{11,\,0.025} = 2.201, this is far beyond the critical value; p \approx 6.43 \times 10^{-6}. Strong evidence the drug lowers blood pressure.

Effect size matters as much as the verdict:

$d = \frac{\bar d}{s_d} = \frac{6.0}{2.594} \approx 2.31$

A Cohen's d above 0.8 is conventionally "large", so 2.3 is very large — and a 6 mmHg average reduction is clinically meaningful, not just statistically detectable.

Compare with the wrong analysis. Treating these as two independent groups gives t = 2.5367 and p = 0.0200 (Welch) — still significant, but with a statistic more than three times smaller. The patient-to-patient variation (138 to 160 mmHg) dominates the two-sample SE, while pairing cancels it entirely. Same data, dramatically weaker conclusion.

Doing it in Python

All three tests, with SciPy:

import numpy as np
from scipy import stats

before = np.array([145, 152, 138, 160, 149, 155, 142, 158, 147, 151, 144, 156])
after = np.array([138, 145, 135, 150, 144, 147, 140, 149, 143, 145, 141, 148])

# Paired -- the correct analysis
paired = stats.ttest_rel(before, after)
print(f"paired      : t = {paired.statistic:7.4f}  p = {paired.pvalue:.3e}")

# Same numbers treated as independent groups -- the WRONG analysis
indep = stats.ttest_ind(before, after, equal_var=False)
print(f"independent : t = {indep.statistic:7.4f}  p = {indep.pvalue:.4f}")

# Equivalent to a one-sample test on the differences
d = before - after
one = stats.ttest_1samp(d, 0)
print(f"one-sample d: t = {one.statistic:7.4f}  p = {one.pvalue:.3e}")

print(f"\nmean difference {d.mean():.2f}, sd {d.std(ddof=1):.4f}")
print(f"Cohen's d      {d.mean()/d.std(ddof=1):.4f}")

Why Welch should be the default — the pooled test's error rate breaks down with unequal variances and unequal group sizes:

import numpy as np
from scipy import stats

rng = np.random.default_rng(0)
trials = 2_000

print("Both groups have the SAME mean, so every rejection is a false positive.")
print(f"\n{'n1':>4} {'n2':>4} {'sd1':>5} {'sd2':>5} {'pooled':>9} {'Welch':>9}")
for n1, n2, s1, s2 in [(10, 10, 1, 1), (10, 10, 1, 4), (5, 25, 1, 4), (25, 5, 1, 4)]:
    pooled_hits = welch_hits = 0
    for _ in range(trials):
        a = rng.normal(0, s1, n1)
        b = rng.normal(0, s2, n2)
        pooled_hits += stats.ttest_ind(a, b, equal_var=True).pvalue < 0.05
        welch_hits += stats.ttest_ind(a, b, equal_var=False).pvalue < 0.05
    print(f"{n1:>4} {n2:>4} {s1:>5} {s2:>5} {pooled_hits/trials:>9.4f} {welch_hits/trials:>9.4f}")

print("\nNominal rate is 0.05. Pooled is badly off when variances and sizes both differ.")

And how much precision pairing buys, when the pairing is real:

import numpy as np
from scipy import stats

rng = np.random.default_rng(1)
n, trials, effect = 15, 3_000, 1.0

paired_hits = indep_hits = 0
for _ in range(trials):
    subject = rng.normal(0, 8, n)              # large between-subject variation
    before = subject + rng.normal(0, 1, n)
    after = subject - effect + rng.normal(0, 1, n)
    paired_hits += stats.ttest_rel(before, after).pvalue < 0.05
    indep_hits += stats.ttest_ind(before, after, equal_var=False).pvalue < 0.05

print(f"true effect = {effect}, between-subject sd = 8")
print(f"  paired test power     : {paired_hits/trials:.4f}")
print(f"  independent test power: {indep_hits/trials:.4f}")
print("\nIgnoring the pairing throws away nearly all the power.")

Your turn

1. n = 16, \bar x = 52, s = 8. Test H_0: \mu = 50 two-sided at \alpha = 0.05.

2. Which test: comparing exam scores of 30 students before and after a course?

3. Which test: comparing heights of 40 men and 35 women?

Solutions

$\text{SE} = \frac{8}{\sqrt{16}} = 2, \qquad t = \frac{52 - 50}{2} = 1.0$

With df = 15, the critical value is t_{15,\,0.025} = 2.131. Since |1.0| < 2.131, fail to reject. The p-value is about 0.333.

Note what this does not say: it doesn't show \mu = 50. The 95% CI is 52 \pm 2.131(2) = (47.74, 56.26), which is consistent with a wide range of values. With n = 16 this study simply lacks the precision to distinguish them.

2. Paired t-test.

The same 30 students are measured twice, so the observations come in matched pairs. Compute each student's improvement and test whether the mean improvement is zero.

Using an independent-samples test here would treat the two sets of scores as unrelated groups, discarding the pairing — and since students differ substantially from one another, that variation would swamp the effect.

3. Two-sample t-test, Welch's version.

The two groups contain different people, so there's no pairing available. Use Welch rather than pooled because there's no reason to assume equal variances, the group sizes differ (40 vs 35), and Welch costs almost nothing when the variances happen to match.

Check yourself in code

Run the blood-pressure example as a paired test and confirm it's equivalent to a one-sample test on the differences — and much stronger than the incorrect independent-samples analysis.

Print exactly this:

paired t 8.0135
one-sample t 8.0135
equivalent: True
independent t 2.5367

Round every statistic to 4 decimal places.

import numpy as np
from scipy import stats

before = np.array([145, 152, 138, 160, 149, 155, 142, 158, 147, 151, 144, 156])
after = np.array([138, 145, 135, 150, 144, 147, 140, 149, 143, 145, 141, 148])

paired = stats.ttest_rel(before, after).statistic
print("paired t", round(paired, 4))

# Run a one-sample test on the differences, confirm the two match,
# then run the (incorrect) independent-samples test with equal_var=False.

import numpy as np
from scipy import stats

before = np.array([145, 152, 138, 160, 149, 155, 142, 158, 147, 151, 144, 156])
after = np.array([138, 145, 135, 150, 144, 147, 140, 149, 143, 145, 141, 148])

paired = stats.ttest_rel(before, after).statistic
print("paired t", round(paired, 4))

one = stats.ttest_1samp(before - after, 0).statistic
print("one-sample t", round(one, 4))
print("equivalent:", round(paired, 4) == round(one, 4))

indep = stats.ttest_ind(before, after, equal_var=False).statistic
print("independent t", round(indep, 4))

The t-test compares means when \sigma is unknown, paying for that ignorance with heavier tails. Use the one-sample version against a fixed value, Welch's two-sample version for independent groups, and the paired version whenever the observations are genuinely matched — because pairing cancels between-subject variation and is usually where the power comes from.

Next: tests for counts and categories, where the \chi^2 distribution takes over.

36. t-tests