Hypotheses, errors and power · Probability & Statistics

Estimation asks "what is \theta?" Hypothesis testing asks a different question: "is the data compatible with a specific claim about \theta?"

It's a decision procedure, and like any decision procedure it can be wrong in two distinct ways.

The two hypotheses

Null hypothesis H_0 — the default, the status quo, the "nothing interesting is happening" claim. The drug has no effect; the coin is fair; the two groups are the same.

Alternative hypothesis H_1 — what you'd conclude instead.

The asymmetry between them is deliberate and is the whole design of the method:

H_0 is assumed true until the evidence against it is strong.
We never accept H_0 — we either reject it or fail to reject it.

That awkward phrasing is doing real work. Failing to reject means the evidence wasn't strong enough, which is not the same as showing H_0 is true. A study of 5 patients will fail to reject almost anything.

The analogy is a criminal trial: the defendant is presumed innocent, and a verdict of "not guilty" means the prosecution didn't meet its burden — not that innocence was proved.

Hypotheses are about parameters, never about statistics. H_0: \mu = 100 is a hypothesis; "H_0: \bar x = 100" is meaningless, since \bar x is something you observe rather than something that might be true.

The two errors

	H_0 true	H_0 false
Reject H_0	Type I error (\alpha)	correct ✓
Fail to reject	correct ✓	Type II error (\beta)

Type I error — rejecting a true null. A false positive. Convicting the innocent. Its probability is \alpha, the significance level, which you choose (conventionally 0.05).

Type II error — failing to reject a false null. A false negative. Letting the guilty go. Its probability is \beta, which you do not directly control.

Power is the complement of the Type II error:

$\text{Power} = 1 - \beta = P(\text{reject } H_0 \mid H_0 \text{ false})$

The probability of detecting a real effect. Conventionally you aim for 0.80 or better.

The trade-off

\alpha and \beta pull against each other. Demanding stronger evidence (smaller \alpha) means rejecting less often, which means missing more real effects (larger \beta).

You could drive \alpha to zero by never rejecting anything — and have zero power. You could drive \beta to zero by always rejecting — and be wrong every time H_0 is true.

The only way to improve both at once is more data. That's what a larger n buys: the sampling distributions under H_0 and H_1 separate, and both error rates can fall together.

What drives power

Power depends on four things, and knowing which is which is most of study design:

$\text{Power} \;\uparrow\; \text{as} \begin{cases} \text{effect size } |\mu_1 - \mu_0| \;\uparrow & \text{bigger effects are easier to see} \\ n \;\uparrow & \text{more data, less noise} \\ \sigma \;\downarrow & \text{less noise to begin with} \\ \alpha \;\uparrow & \text{a lower bar to clear} \end{cases}$

These combine into a single quantity. For a one-sample z-test:

$\text{Power} = \Phi\!\left(\frac{|\mu_1 - \mu_0|\sqrt n}{\sigma} - z_{1-\alpha}\right)$

The standardised effect size d = \frac{|\mu_1 - \mu_0|}{\sigma} (Cohen's d) is what matters, not the raw difference — a 5-unit difference is enormous if \sigma = 1 and invisible if \sigma = 100.

Two failure modes

Underpowered studies are the more discussed problem. With power 0.3, you miss 70% of real effects. Worse, the effects you do detect are systematically overestimated — only unusually large sample results clear the bar, so published estimates from small studies are inflated. This is the "winner's curse", and it is a major driver of the replication crisis.

Overpowered studies have the opposite issue. With n = 10^6, a difference of 0.001 becomes statistically significant while being of no practical consequence whatsoever. Statistical significance is not importance, and at large n the two come apart badly. Always report the effect size, not just the verdict.

Worked example

H_0: \mu = 100 vs H_1: \mu > 100, with \sigma = 15, n = 25, \alpha = 0.05. Find the rejection region and the power if \mu = 105.

The rejection region. Under H_0, \bar X \sim N(100, 15^2/25) = N(100, 9), so SE = 3. A one-sided test at \alpha = 0.05 uses z = 1.645:

$\text{reject if } \bar X > 100 + 1.645(3) = 104.94$

The power. If the truth is \mu = 105, then \bar X \sim N(105, 9), and we ask how often it exceeds the same cutoff:

$z = \frac{104.94 - 105}{3} = -0.02$

$\text{Power} = P(Z > -0.02) = 0.508$

About 51% — barely better than a coin flip at detecting a real 5-point effect.

How much data for 80% power? Solve

$\frac{5\sqrt n}{15} - 1.645 = 0.842 \implies \sqrt n = \frac{15 \times 2.487}{5} = 7.46 \implies n \ge 56$

So 56 observations, more than double. This is the standard power calculation, and it's why it must be done before collecting data, not after.

Doing it in Python

Power computed directly, and checked by simulation:

import numpy as np
from scipy.stats import norm

mu0, mu1, sigma, n, alpha = 100.0, 105.0, 15.0, 25, 0.05
se = sigma / np.sqrt(n)

crit = mu0 + norm.ppf(1 - alpha) * se
power = 1 - norm.cdf(crit, loc=mu1, scale=se)

print(f"standard error   : {se:.4f}")
print(f"rejection cutoff : {crit:.4f}")
print(f"theoretical power: {power:.4f}")

rng = np.random.default_rng(0)
sims = rng.normal(mu1, sigma, size=(200_000, n)).mean(axis=1)
print(f"simulated power  : {(sims > crit).mean():.4f}")

Confirming the Type I error rate is what you asked for:

import numpy as np
from scipy.stats import norm

rng = np.random.default_rng(1)
mu0, sigma, n = 100.0, 15.0, 25
se = sigma / np.sqrt(n)

print(f"{'alpha':>8} {'simulated type I':>18}")
for alpha in (0.10, 0.05, 0.01, 0.001):
    crit = mu0 + norm.ppf(1 - alpha) * se
    sims = rng.normal(mu0, sigma, size=(200_000, n)).mean(axis=1)   # H0 IS true
    print(f"{alpha:>8} {(sims > crit).mean():>18.4f}")

print("\nThe Type I rate is exactly what you set it to -- it's a design choice.")

The four drivers of power, one at a time:

import numpy as np
from scipy.stats import norm

def power(mu0, mu1, sigma, n, alpha=0.05):
    se = sigma / np.sqrt(n)
    crit = mu0 + norm.ppf(1 - alpha) * se
    return 1 - norm.cdf(crit, loc=mu1, scale=se)

print("effect size (n=25, sigma=15):")
for mu1 in (101, 103, 105, 110, 115):
    print(f"   mu1 = {mu1:>4}: power {power(100, mu1, 15, 25):.4f}")

print("\nsample size (effect = 5, sigma = 15):")
for n in (10, 25, 56, 100, 400):
    print(f"   n = {n:>4}: power {power(100, 105, 15, n):.4f}")

print("\nnoise (effect = 5, n = 25):")
for sigma in (5, 10, 15, 30):
    print(f"   sigma = {sigma:>3}: power {power(100, 105, sigma, 25):.4f}")

print("\nalpha (effect = 5, n = 25, sigma = 15):")
for a in (0.10, 0.05, 0.01):
    print(f"   alpha = {a:>5}: power {power(100, 105, 15, 25, a):.4f}")

And the winner's curse — why underpowered studies overstate effects:

import numpy as np
from scipy.stats import norm

rng = np.random.default_rng(2)
mu0, true_mu, sigma, alpha = 100.0, 102.0, 15.0, 0.05
true_effect = true_mu - mu0

print(f"True effect is {true_effect}. Among studies that reach significance:\n")
print(f"{'n':>6} {'power':>8} {'mean published effect':>24}")
for n in (10, 25, 100, 400):
    se = sigma / np.sqrt(n)
    crit = mu0 + norm.ppf(1 - alpha) * se
    xbar = rng.normal(true_mu, se, size=200_000)
    sig = xbar > crit
    print(f"{n:>6} {sig.mean():>8.4f} {(xbar[sig] - mu0).mean():>24.4f}")

print("\nLow power inflates the effects that get published -- badly.")

Your turn

1. A test has \alpha = 0.05 and power 0.80. What are the probabilities of each error type?

2. Which error is worse: a fire alarm that goes off with no fire, or one that stays silent during a fire?

3. A study finds p = 0.04 with n = 10{,}000 and an effect of 0.02 units. Is it important?

Solutions

$\alpha = P(\text{Type I}) = 0.05, \qquad \beta = P(\text{Type II}) = 1 - \text{power} = 0.20$

Note these are conditional probabilities on different conditions: \alpha assumes H_0 is true, \beta assumes it's false. They don't sum to anything meaningful, and neither one tells you P(H_0 \text{ true}) — that's a Bayesian question (§7).

2. It depends entirely on the costs, which is the real point.

A false alarm (Type I) causes evacuation, disruption, and eventually people ignoring alarms. A missed fire (Type II) can kill people.

Here the Type II error is far worse, so you'd set a lenient threshold — a high \alpha — accepting many false alarms to make misses very unlikely. Smoke detectors are deliberately tuned this way, which is why burnt toast sets them off.

Reverse the costs and you reverse the design. A test that would send someone to prison should have a very small \alpha. The 0.05 convention is a convention, not a principle — the right \alpha depends on the relative cost of the two mistakes.

3. Statistically significant, almost certainly not important.

With n = 10{,}000 the standard error is tiny, so even a trivial effect clears the significance bar. The p-value tells you the effect is probably not exactly zero. It says nothing about whether 0.02 units matters.

The questions to ask: what's the confidence interval (it's probably something like (0.001, 0.039) — consistent with an effect near zero)? And what's the smallest effect that would change a decision? If it takes 0.5 units to matter, this result is a precisely-measured irrelevance.

This is the overpowered-study failure mode, and it's why journals increasingly require effect sizes and intervals rather than p-values alone.

Check yourself in code

Compute the power of a one-sided z-test analytically and confirm it by simulation.

Print exactly this:

cutoff 104.9346
theoretical power 0.5087
simulated power 0.5085
agree: True

Use \mu_0 = 100, \mu_1 = 105, \sigma = 15, n = 25, \alpha = 0.05, default_rng(0) and 200000 simulated samples. Round the cutoff to 4 decimal places and both powers to 4. Report agree: True if they differ by less than 0.01.

import numpy as np
from scipy.stats import norm

mu0, mu1, sigma, n, alpha = 100.0, 105.0, 15.0, 25, 0.05
se = sigma / np.sqrt(n)

crit = mu0 + norm.ppf(1 - alpha) * se
print("cutoff", round(crit, 4))

# Compute the theoretical power, simulate 200000 samples drawn under mu1
# with default_rng(0), and compare the two.

import numpy as np
from scipy.stats import norm

mu0, mu1, sigma, n, alpha = 100.0, 105.0, 15.0, 25, 0.05
se = sigma / np.sqrt(n)

crit = mu0 + norm.ppf(1 - alpha) * se
print("cutoff", round(crit, 4))

theoretical = 1 - norm.cdf(crit, loc=mu1, scale=se)
print("theoretical power", round(theoretical, 4))

rng = np.random.default_rng(0)
sims = rng.normal(mu1, sigma, size=(200_000, n)).mean(axis=1)
simulated = (sims > crit).mean()
print("simulated power", round(simulated, 4))
print("agree:", bool(abs(theoretical - simulated) < 0.01))

A test presumes H_0 and asks whether the evidence is strong enough to abandon it. Type I error is rejecting a true null, at a rate \alpha you choose; Type II is missing a real effect, at a rate \beta you control only through design. Power is 1 - \beta, and it rises with effect size and n, falls with noise. Underpowered studies miss real effects and inflate the ones they find; overpowered studies find effects too small to care about.

Next: the p-value — what it measures, and the four things it is routinely mistaken for.

33. Hypotheses, errors and power