F-tests and ANOVA · Probability & Statistics

You have three teaching methods and want to know whether they differ. Why not just run three t-tests — A vs B, A vs C, B vs C?

Because each carries a 5% false-positive rate, and three of them push the family-wise rate to about 14%. With five groups you'd need ten comparisons and the rate climbs past 40%.

ANOVA answers the question with a single test.

The idea: partition the variation

Analysis of variance compares means by comparing variances — which sounds backwards until you see the decomposition.

Total variation in the data splits into two parts:

$\underbrace{\sum_{ij}(x_{ij} - \bar x)^2}_{SS_{\text{total}}} = \underbrace{\sum_i n_i(\bar x_i - \bar x)^2}_{SS_{\text{between}}} + \underbrace{\sum_{ij}(x_{ij} - \bar x_i)^2}_{SS_{\text{within}}}$

Between-groups: how far the group means are from the overall mean.
Within-groups: how much observations scatter inside their own group — pure noise.

If the groups genuinely differ, between-group variation will be large relative to the noise. If they don't, both estimate the same \sigma^2 and their ratio should be near 1.

The F statistic

Convert sums of squares to mean squares by dividing by degrees of freedom, then take the ratio:

$F = \frac{MS_{\text{between}}}{MS_{\text{within}}} = \frac{SS_{\text{between}}/(k-1)}{SS_{\text{within}}/(N-k)}$

with k groups and N observations total.

Under H_0: \mu_1 = \cdots = \mu_k, both mean squares are unbiased estimates of \sigma^2, so

$F \sim F_{k-1,\;N-k}$

Under H_1, the numerator is inflated by the real differences while the denominator isn't — so the test is one-sided in the upper tail. Only large F is evidence.

The degrees of freedom: k-1 because the k group means are constrained by the overall mean; N-k because each group loses one to its own mean.

The ANOVA table

Source	SS	df	MS	F
Between	SS_B	k-1	SS_B/(k-1)	MS_B/MS_W
Within	SS_W	N-k	SS_W/(N-k)
Total	SS_T	N-1

Assumptions

Independent observations, within and between groups.
Normality within each group — though ANOVA is fairly robust to this at moderate n, thanks to the CLT.
Equal variances (homoscedasticity). This one matters more, especially with unequal group sizes. Check it with Levene's test, or use Welch's ANOVA if it fails.

What a significant F does not tell you

Which groups differ. A significant F says "not all means are equal" and stops there. With five groups it won't say whether one is odd or all five are distinct.

For that you need post-hoc tests, which control for the multiple comparisons you're now making:

Tukey's HSD — all pairwise comparisons, family-wise rate controlled.
Bonferroni — simple, conservative.
Dunnett's — comparing several treatments against one control only.

Running plain t-tests after a significant F reintroduces exactly the problem ANOVA was meant to avoid.

Two-way ANOVA

With two factors — say teaching method and class size — you can test three things at once:

Main effect of factor A
Main effect of factor B
Interaction — does the effect of A depend on the level of B?

The interaction is usually the interesting part, and it's invisible to separate one-way analyses. A drug that helps young patients and harms old ones may show no main effect at all while having a large interaction.

When an interaction is significant, interpret main effects with care — "the average effect of the drug" is close to meaningless if the sign flips between subgroups.

The connection to t and F

For two groups, ANOVA and the pooled t-test are the same test:

$F_{1,\,N-2} = t^2_{N-2}$

Not approximately — exactly. The F distribution with 1 numerator degree of freedom is the square of a t. So ANOVA is the natural generalisation of the two-sample t-test to more than two groups.

Worked example

Three teaching methods, 5 students each:

Method A	85	88	82	90	85	\bar x_A = 86
Method B	78	75	80	77	80	\bar x_B = 78
Method C	92	89	94	91	89	\bar x_C = 91

Overall mean: \bar x = (86 + 78 + 91)/3 = 85.

Between-group SS:

$SS_B = 5\big[(86-85)^2 + (78-85)^2 + (91-85)^2\big] = 5[1 + 49 + 36] = 430$

Within-group SS, summing squared deviations inside each group:

$SS_W = 38 + 18 + 18 = 74$

The table:

$MS_B = \frac{430}{2} = 215, \qquad MS_W = \frac{74}{12} \approx 6.167$

$F = \frac{215}{6.167} \approx 34.86$

With df = (2, 12), the critical value is F_{2,12,\,0.05} = 3.89. Since 34.86 \gg 3.89, reject — the methods differ (p \approx 1.0 \times 10^{-5}).

Effect size. Report \eta^2, the proportion of variance explained:

$\eta^2 = \frac{SS_B}{SS_T} = \frac{430}{504} \approx 0.853$

85% of the variation in scores is between methods rather than within them — a very large effect.

And now the follow-up. The F says the methods aren't all equal. Looking at the means, C (91) > A (86) > B (78), and Tukey's HSD would confirm all three pairwise differences are significant. Without that step, "the methods differ" is as far as you can honestly go.

Doing it in Python

The one-way ANOVA, and the decomposition behind it:

import numpy as np
from scipy.stats import f_oneway, f

A = np.array([85, 88, 82, 90, 85])
B = np.array([78, 75, 80, 77, 80])
C = np.array([92, 89, 94, 91, 89])

result = f_oneway(A, B, C)
print(f"F = {result.statistic:.4f}   p = {result.pvalue:.3e}")

# Build it by hand
groups = [A, B, C]
all_data = np.concatenate(groups)
grand = all_data.mean()
k, N = len(groups), len(all_data)

ss_between = sum(len(g) * (g.mean() - grand) ** 2 for g in groups)
ss_within = sum(((g - g.mean()) ** 2).sum() for g in groups)
ss_total = ((all_data - grand) ** 2).sum()

print(f"\nSS between {ss_between:8.2f}  df {k-1}")
print(f"SS within  {ss_within:8.2f}  df {N-k}")
print(f"SS total   {ss_total:8.2f}  df {N-1}   (check: {ss_between + ss_within:.2f})")

F = (ss_between / (k - 1)) / (ss_within / (N - k))
print(f"\nF = {F:.4f}   critical {f.ppf(0.95, k-1, N-k):.4f}")
print(f"eta^2 = {ss_between / ss_total:.4f}")

Why not run multiple t-tests — measured:

import numpy as np
from scipy.stats import ttest_ind, f_oneway

rng = np.random.default_rng(0)
trials, n = 5_000, 10

for k in (2, 3, 5):
    t_hits = anova_hits = 0
    for _ in range(trials):
        groups = [rng.normal(0, 1, n) for _ in range(k)]      # all means EQUAL
        pairs = [(i, j) for i in range(k) for j in range(i + 1, k)]
        any_t = any(ttest_ind(groups[i], groups[j]).pvalue < 0.05 for i, j in pairs)
        t_hits += any_t
        anova_hits += f_oneway(*groups).pvalue < 0.05
    print(f"k={k} groups ({len(pairs)} pairwise tests): "
          f"any t-test significant {t_hits/trials:.4f}, ANOVA {anova_hits/trials:.4f}")

print("\nANOVA holds at 0.05; the pairwise approach inflates badly as k grows.")

The exact F = t^2 identity for two groups:

import numpy as np
from scipy.stats import ttest_ind, f_oneway

rng = np.random.default_rng(1)
a = rng.normal(10, 2, 15)
b = rng.normal(11, 2, 18)

t_res = ttest_ind(a, b, equal_var=True)      # pooled, to match ANOVA
f_res = f_oneway(a, b)

print(f"t          = {t_res.statistic:.6f}")
print(f"t^2        = {t_res.statistic**2:.6f}")
print(f"F          = {f_res.statistic:.6f}")
print(f"equal?       {np.isclose(t_res.statistic**2, f_res.statistic)}")
print(f"\np-values:  t {t_res.pvalue:.6f}   F {f_res.pvalue:.6f}")

And post-hoc testing, which the F alone can't give you:

import numpy as np
from itertools import combinations
from scipy.stats import ttest_ind, f_oneway

A = np.array([85, 88, 82, 90, 85])
B = np.array([78, 75, 80, 77, 80])
C = np.array([92, 89, 94, 91, 89])
names, groups = ["A", "B", "C"], [A, B, C]

print(f"omnibus F p-value: {f_oneway(*groups).pvalue:.3e}\n")
print("pairwise, with a Bonferroni correction:")
pairs = list(combinations(range(3), 2))
alpha_adj = 0.05 / len(pairs)
for i, j in pairs:
    p = ttest_ind(groups[i], groups[j]).pvalue
    verdict = "differ" if p < alpha_adj else "no evidence"
    print(f"  {names[i]} vs {names[j]}: p = {p:.5f}  (threshold {alpha_adj:.4f})  {verdict}")

Your turn

1. 4 groups, 10 observations each. What are the degrees of freedom for the F test?

2. SS_B = 120, SS_W = 240, k = 3, N = 30. Compute F and \eta^2.

3. ANOVA gives p = 0.02. What can you conclude?

Solutions

1. k = 4 groups, N = 40 observations.

$df_{\text{between}} = k - 1 = 3, \qquad df_{\text{within}} = N - k = 40 - 4 = 36$

So the test uses F_{3,\,36}.

$MS_B = \frac{120}{3-1} = 60, \qquad MS_W = \frac{240}{30-3} = \frac{240}{27} \approx 8.889$

$F = \frac{60}{8.889} \approx 6.75$

$\eta^2 = \frac{SS_B}{SS_B + SS_W} = \frac{120}{360} = 0.333$

With F_{2,27,\,0.05} \approx 3.35, this is significant. About a third of the total variation is between groups.

3. Conclude: the group means are not all equal. Reject H_0 at the 5% level.

Do not conclude:

Which groups differ. The F test is an omnibus test — it detects that a difference exists somewhere. You need post-hoc comparisons for the pattern.
All groups differ from each other. It's entirely possible that one group is unusual and the rest are indistinguishable.
The differences are large. As always, p depends on sample size; report \eta^2 and the group means.

The disciplined sequence is: significant F → post-hoc tests with a correction → report effect sizes and intervals.

Check yourself in code

Run the one-way ANOVA on the three teaching methods, verify the sums of squares decompose correctly, and compute the effect size.

Print exactly this:

F 34.8649
p 0.0
SS decomposes: True
eta squared 0.8532

Round F to 4 decimal places, the p-value to 4, and \eta^2 to 4. Report SS decomposes: True if SS_B + SS_W equals SS_T to within 10^{-9}.

import numpy as np
from scipy.stats import f_oneway

A = np.array([85, 88, 82, 90, 85])
B = np.array([78, 75, 80, 77, 80])
C = np.array([92, 89, 94, 91, 89])
groups = [A, B, C]

result = f_oneway(*groups)
print("F", round(result.statistic, 4))
print("p", round(result.pvalue, 4))

# Compute SS between, SS within and SS total, confirm they decompose,
# and print eta squared = SS_between / SS_total.

import numpy as np
from scipy.stats import f_oneway

A = np.array([85, 88, 82, 90, 85])
B = np.array([78, 75, 80, 77, 80])
C = np.array([92, 89, 94, 91, 89])
groups = [A, B, C]

result = f_oneway(*groups)
print("F", round(result.statistic, 4))
print("p", round(result.pvalue, 4))

all_data = np.concatenate(groups)
grand = all_data.mean()
ss_between = sum(len(g) * (g.mean() - grand) ** 2 for g in groups)
ss_within = sum(((g - g.mean()) ** 2).sum() for g in groups)
ss_total = ((all_data - grand) ** 2).sum()

print("SS decomposes:", bool(abs(ss_between + ss_within - ss_total) < 1e-9))
print("eta squared", round(ss_between / ss_total, 4))

ANOVA tests whether several means are equal by splitting total variation into between-group and within-group parts and taking their ratio. Large F means the group differences outweigh the noise. It avoids the multiple-comparisons inflation that repeated t-tests would cause — but it only tells you that some difference exists, so post-hoc tests and effect sizes are not optional extras.

Next: what to do when the Normality assumptions behind all of this don't hold.

38. F-tests and ANOVA