p-values and significance · Probability & Statistics

The p-value is the most reported number in empirical science and the most misunderstood. This lesson defines it precisely, then works through what it is not — because nearly every misuse is one of four specific confusions.

The definition

$p = P\big(\text{a test statistic at least as extreme as the one observed} \mid H_0 \text{ true}\big)$

Read the conditioning carefully. We assume H_0 is true and ask how surprising the data is under that assumption.

A small p means: if the null were true, data like this would be rare. That is evidence against H_0 — either something unlikely happened, or the assumption was wrong.

"At least as extreme" matters. The p-value includes the observed result and everything more extreme, because any single outcome from a continuous distribution has probability zero. What's informative is the tail.

Computing one

Standardise, then take the appropriate tail:

$z = \frac{\bar x - \mu_0}{\sigma/\sqrt n}$

Alternative	p-value
H_1: \mu > \mu_0	P(Z > z)
H_1: \mu < \mu_0	P(Z < z)
H_1: \mu \ne \mu_0	2 \times P(Z > \lvert z \rvert)

The factor of 2 for a two-sided test is because "extreme" now means extreme in either direction.

Decide the direction before seeing the data. Switching to a one-sided test after noticing which way the result went silently doubles your Type I error rate.

The relationship to \alpha

$\text{reject } H_0 \iff p \le \alpha$

The p-value is the smallest \alpha at which you'd reject. It's a continuous measure of evidence, which the reject/don't-reject verdict throws away — which is why reporting the value itself is better practice than reporting "significant".

There's nothing special about 0.05. Fisher suggested it as a rough convenience; it became a convention through repetition. Particle physics demands about 3 \times 10^{-7} (the "5 sigma" rule) because it runs enormous numbers of comparisons.

The four things a p-value is not

1. Not P(H_0 \text{ true} \mid \text{data})

This is the most common error. The p-value conditions on H_0 being true and gives the probability of the data. The thing people want is the reverse conditional.

$p = P(\text{data} \mid H_0) \;\;\ne\;\; P(H_0 \mid \text{data})$

Getting from one to the other requires Bayes' theorem and a prior — the exact flip from §0. And the two can differ enormously: with a plausible prior, a result at p = 0.05 often corresponds to a posterior probability of the null of 20–30%, not 5%.

2. Not the probability the result was chance

Same confusion, differently worded. "There's a 5% chance this was a fluke" is a statement about P(H_0 \mid \text{data}), which the p-value does not provide.

3. Not a measure of effect size

p depends on both the effect and the sample size. A huge n makes a trivial effect significant; a small n makes a large effect non-significant. A tiny p means "probably not exactly zero", not "big".

4. Not evidence for H_0 when large

p = 0.6 means the data is compatible with H_0. It's also compatible with many alternatives, especially in a small study. Absence of evidence is not evidence of absence — to claim no effect you need an equivalence test or a confidence interval tight enough to exclude anything meaningful.

The multiple comparisons problem

Test 20 independent true nulls at \alpha = 0.05. The chance of at least one false positive is

$1 - (0.95)^{20} = 0.64$

Sixty-four percent. Test 100 and it's 99.4%. Significant results are guaranteed if you look at enough things.

This is where a large fraction of irreproducible findings come from, especially in combination with:

p-hacking — trying analyses until something crosses 0.05: dropping outliers, adding covariates, splitting subgroups, stopping data collection when the result looks good. Each choice is defensible in isolation; collectively they invalidate the p-value entirely, because the "at least as extreme" calculation assumed one pre-specified analysis.

Corrections:

Bonferroni: test at \alpha/m for m tests. Controls the chance of any false positive; conservative and loses power.
Benjamini–Hochberg: controls the false discovery rate — the expected proportion of rejections that are false. Much better power, and the standard choice when m is large.

Pre-registration — committing to the analysis before seeing the data — is the structural fix.

Worked example

A coin lands heads 60 times in 100 flips. Test H_0: p = 0.5 two-sided.

Under H_0, the count has mean 50 and \sigma = \sqrt{100 \times 0.25} = 5.

$z = \frac{60 - 50}{5} = 2.0$

$p = 2 \times P(Z > 2) = 2(0.0228) = 0.0455$

At \alpha = 0.05, reject — the coin is probably not fair.

Now read that carefully. It does not mean there's a 4.55% chance the coin is fair. It means: if the coin were fair, we'd see a result this extreme about 4.6% of the time.

And the effect size deserves attention too. The estimate is \hat p = 0.60 with a 95% CI of roughly (0.50, 0.70) — the interval barely excludes 0.5, so the evidence is genuinely marginal. A result of p = 0.046 is not strong evidence; it's the weakest thing that clears the conventional bar.

Compare: 600 heads in 1,000 flips gives the same \hat p = 0.60 but z = 6.3 and p \approx 3 \times 10^{-10}. Same effect, vastly different p — which is exactly point 3 above.

Doing it in Python

The coin test, exactly and by approximation:

import numpy as np
from scipy.stats import norm, binomtest

k, n = 60, 100

z = (k - n * 0.5) / np.sqrt(n * 0.25)
p_normal = 2 * (1 - norm.cdf(abs(z)))
p_exact = binomtest(k, n, 0.5, alternative="two-sided").pvalue

print(f"z statistic        : {z:.4f}")
print(f"normal approx p    : {p_normal:.6f}")
print(f"exact binomial p   : {p_exact:.6f}")

# Same proportion, ten times the data
k2, n2 = 600, 1000
z2 = (k2 - n2 * 0.5) / np.sqrt(n2 * 0.25)
print(f"\nsame p_hat = 0.6 at n = 1000: z = {z2:.4f}, p = {2*(1-norm.cdf(abs(z2))):.3e}")
print("-> identical effect size, wildly different p-value")

Under a true null, p-values are uniform — that's the property that makes the whole framework work:

import numpy as np
from scipy.stats import norm, kstest

rng = np.random.default_rng(0)
n, trials = 30, 100_000

x = rng.normal(0, 1, size=(trials, n))          # H0: mu = 0 is TRUE
z = x.mean(axis=1) / (1 / np.sqrt(n))
p = 2 * (1 - norm.cdf(np.abs(z)))

print("Under a true H0, p-values are Uniform(0,1):")
print("  KS test against uniform, p =", round(kstest(p, "uniform").pvalue, 4))
for thresh in (0.01, 0.05, 0.10, 0.50):
    print(f"  P(p <= {thresh:<4}) = {(p <= thresh).mean():.4f}   (should be {thresh})")

The multiple comparisons problem, and two corrections:

import numpy as np
from scipy.stats import norm

rng = np.random.default_rng(1)
m, trials, n, alpha = 20, 20_000, 30, 0.05

any_sig = 0
for _ in range(trials):
    x = rng.normal(0, 1, size=(m, n))           # every null is TRUE
    z = x.mean(axis=1) * np.sqrt(n)
    p = 2 * (1 - norm.cdf(np.abs(z)))
    any_sig += (p <= alpha).any()

print(f"Testing {m} true nulls at alpha = {alpha}:")
print(f"  P(at least one 'significant') = {any_sig/trials:.4f}")
print(f"  theory 1 - 0.95^{m}            = {1 - 0.95**m:.4f}")

import numpy as np
from scipy.stats import norm

rng = np.random.default_rng(2)
m, trials, n, alpha = 20, 20_000, 30, 0.05

def family_error(correction):
    hits = 0
    for _ in range(trials):
        x = rng.normal(0, 1, size=(m, n))
        p = np.sort(2 * (1 - norm.cdf(np.abs(x.mean(axis=1) * np.sqrt(n)))))
        if correction == "none":
            hits += (p <= alpha).any()
        elif correction == "bonferroni":
            hits += (p <= alpha / m).any()
        elif correction == "bh":
            thresh = alpha * np.arange(1, m + 1) / m
            hits += (p <= thresh).any()
    return hits / trials

for c in ("none", "bonferroni", "bh"):
    print(f"{c:>12}: family-wise false positive rate {family_error(c):.4f}")

Your turn

1. p = 0.03 with \alpha = 0.05. What do you conclude? What don't you conclude?

2. You run 40 tests, all nulls true, at \alpha = 0.05. How many significant results do you expect?

3. Study A: p = 0.049. Study B: p = 0.051. How different are they?

Solutions

1. Conclude: reject H_0 at the 5% level. If H_0 were true, data this extreme would occur about 3% of the time.

Do not conclude:

"There's a 3% chance H_0 is true." That's P(H_0 \mid \text{data}), which requires a prior.
"There's a 97% chance H_1 is true." Same error, complemented.
"The effect is large." p says nothing about magnitude — report the estimate and its confidence interval.
"The result will replicate." A study at p = 0.03 has surprisingly low probability of replicating; that probability depends on power and the true effect, not on p.

2. Each test rejects with probability 0.05 under a true null, so the count is Binomial(40, 0.05):

$E[\text{significant}] = 40 \times 0.05 = 2$

Two false positives on average, with $P(\text{at least one}) = 1 - 0.95^{40} = 0.87$. If you ran 40 tests and found 2 significant, you have found exactly nothing.

3. Essentially identical.

The difference between 0.049 and 0.051 is statistical noise. Both say the data is mildly unusual under H_0; neither is strong evidence.

Yet under a strict 0.05 rule, A is "significant" and B is "not" — and in practice A gets published while B goes in a drawer. That dichotomy is an artefact of the threshold, not of the evidence, and it's a large part of why publication bias exists.

The fix is to report p as a continuous measure alongside the effect size and interval, rather than collapsing it to a binary verdict.

Check yourself in code

Confirm the two facts that make p-values interpretable: under a true null they are uniform, and testing many true nulls produces false positives at a predictable rate.

Print exactly this:

P(p <= 0.05) = 0.0498
uniform: True
P(any of 20 significant) = 0.6374
theory = 0.6415

Use default_rng(0), n = 30, 100000 trials for the first part; then default_rng(1), 20000 trials of 20 tests each for the second. Round the first proportion to 4 decimal places, report uniform: True if it is within 0.005 of 0.05, and round both of the last two values to 4.

import numpy as np
from scipy.stats import norm

rng = np.random.default_rng(0)
n, trials = 30, 100_000

x = rng.normal(0, 1, size=(trials, n))
p = 2 * (1 - norm.cdf(np.abs(x.mean(axis=1) * np.sqrt(n))))

rate = (p <= 0.05).mean()
print("P(p <= 0.05) =", round(rate, 4))
print("uniform:", bool(abs(rate - 0.05) < 0.005))

# Now run 20000 trials of 20 tests each (all nulls true) with default_rng(1),
# and report how often at least one is significant, against 1 - 0.95**20.

import numpy as np
from scipy.stats import norm

rng = np.random.default_rng(0)
n, trials = 30, 100_000

x = rng.normal(0, 1, size=(trials, n))
p = 2 * (1 - norm.cdf(np.abs(x.mean(axis=1) * np.sqrt(n))))

rate = (p <= 0.05).mean()
print("P(p <= 0.05) =", round(rate, 4))
print("uniform:", bool(abs(rate - 0.05) < 0.005))

rng2 = np.random.default_rng(1)
m, trials2 = 20, 20_000
hits = 0
for _ in range(trials2):
    y = rng2.normal(0, 1, size=(m, n))
    pv = 2 * (1 - norm.cdf(np.abs(y.mean(axis=1) * np.sqrt(n))))
    hits += (pv <= 0.05).any()

print("P(any of 20 significant) =", round(hits / trials2, 4))
print("theory =", round(1 - 0.95**m, 4))

A p-value is P(\text{data at least this extreme} \mid H_0) — and nothing else. It is not the probability the null is true, not the probability the result was chance, not a measure of effect size, and not evidence for the null when large. Under a true null it is uniform, which is precisely why testing many things guarantees false positives.

Next: the theorem that says which test is best for a given \alpha, and the general recipe it leads to.

34. p-values and significance