Confidence intervals · Probability & Statistics

A point estimate is a single number, and a single number can't express uncertainty. "The mean is 47.3" invites a false sense of precision. A confidence interval attaches a range and a stated reliability.

It is also the most widely misinterpreted object in statistics, so we'll be careful about what it does and doesn't say.

The construction

Start with the CLT (§3). For large n,

$\bar X \approx N\!\left(\mu, \frac{\sigma^2}{n}\right) \implies \frac{\bar X - \mu}{\sigma/\sqrt n} \approx N(0,1)$

A standard Normal lands in (-1.96, 1.96) with probability 0.95, so

$P\left(-1.96 < \frac{\bar X - \mu}{\sigma/\sqrt n} < 1.96\right) \approx 0.95$

Rearrange to isolate \mu:

$P\left(\bar X - 1.96\frac{\sigma}{\sqrt n} < \mu < \bar X + 1.96\frac{\sigma}{\sqrt n}\right) \approx 0.95$

Giving the 95% confidence interval:

$\bar X \pm 1.96\frac{\sigma}{\sqrt n}$

The general form is worth memorising, because nearly every interval in statistics fits it:

$\text{estimate} \;\pm\; (\text{critical value}) \times (\text{standard error})$

Common critical values: 1.645 for 90%, 1.96 for 95%, 2.576 for 99%.

What it actually means

The interval is random; the parameter is not.

\mu is a fixed unknown constant. It doesn't have a distribution and it isn't "probably" anywhere. What varies is \bar X, and therefore the interval.

So the correct reading is about the procedure:

If you repeated this whole process many times — new sample, new interval — about 95% of the intervals produced would contain the true \mu.

The incorrect reading: "there's a 95% probability that \mu is in (4.2, 6.8)."

Once you have computed a specific interval, \mu either is in it or isn't. There's no probability left — the randomness was used up when you drew the sample. The 95% describes the long-run performance of the method, not this particular interval.

If you genuinely want "95% probability that \mu lies in this range", that's a credible interval, and it requires Bayesian machinery (§7). The two often look numerically similar and mean entirely different things.

When \sigma is unknown

Usually it is. Replace \sigma with the sample standard deviation s — and pay for it by widening the interval:

$\bar X \pm t_{n-1,\,\alpha/2}\frac{s}{\sqrt n}$

The t distribution has heavier tails than the Normal, so its critical values are larger. That extra width is the price of not knowing \sigma.

n	t_{n-1,\,0.025}	vs Normal 1.96
5	2.776	42% wider
10	2.262	15% wider
30	2.045	4% wider
100	1.984	1% wider
\infty	1.960	—

By n = 30 the difference is negligible, which is where the "use z for large samples" rule of thumb comes from. Slutsky's theorem (§3) is the formal statement that it vanishes.

The n - 1 degrees of freedom is the same one from the sample variance: estimating \bar X costs you one.

Width, and what controls it

$\text{width} = 2 \times 1.96 \times \frac{\sigma}{\sqrt n}$

More data narrows it, but only as 1/\sqrt n — four times the data for half the width.
Higher confidence widens it. 99% confidence costs 31% more width than 95%.
More variable data widens it, proportionally.

There is no way to get a narrow interval and high confidence and a small sample. That trade-off is the honest content of the method.

A proportion interval

For a proportion, \hat p = \bar X with \operatorname{Var}(X_i) = p(1-p), so

$\hat p \pm 1.96\sqrt{\frac{\hat p(1-\hat p)}{n}}$

This is the Wald interval, and it's the one taught everywhere — but it behaves badly when \hat p is near 0 or 1, or n is small. Its actual coverage can fall well below the nominal 95%.

In the extreme case \hat p = 0, it produces the interval [0, 0] — claiming certainty from data that contains none.

The Wilson interval fixes this and should be preferred; SciPy and statsmodels both provide it. We'll see the coverage difference in code below.

Worked example

A sample of 25 has \bar x = 68 and s = 10. Build a 95% CI for \mu.

\sigma is unknown and n is small, so use t with df = 24:

$t_{24,\,0.025} = 2.064$

$\text{SE} = \frac{s}{\sqrt n} = \frac{10}{5} = 2$

$68 \pm 2.064 \times 2 = 68 \pm 4.13 = (63.87,\; 72.13)$

Interpretation: the procedure that produced (63.87, 72.13) captures the true mean 95% of the time in repeated sampling.

Had we wrongly used z = 1.96 we'd have got (64.08, 71.92) — about 5% narrower, and overstating precision.

To halve the width to \pm 2.06 we would need roughly n = 100.

Doing it in Python

Building intervals both ways:

import numpy as np
from scipy.stats import t, norm

xbar, s, n = 68.0, 10.0, 25
se = s / np.sqrt(n)

t_crit = t.ppf(0.975, df=n - 1)
z_crit = norm.ppf(0.975)

print(f"standard error: {se:.4f}")
print(f"t critical (df={n-1}): {t_crit:.4f}")
print(f"z critical         : {z_crit:.4f}")
print()
print(f"t interval: ({xbar - t_crit*se:.2f}, {xbar + t_crit*se:.2f})")
print(f"z interval: ({xbar - z_crit*se:.2f}, {xbar + z_crit*se:.2f})  <- too narrow")

The claim is about coverage, so let's actually measure it:

import numpy as np
from scipy.stats import t

rng = np.random.default_rng(0)
mu, sigma, n, trials = 50.0, 8.0, 12, 100_000

x = rng.normal(mu, sigma, size=(trials, n))
xbar = x.mean(axis=1)
s = x.std(axis=1, ddof=1)
se = s / np.sqrt(n)

for label, crit in [("t (correct)", t.ppf(0.975, n - 1)), ("z (wrong for small n)", 1.96)]:
    lo, hi = xbar - crit * se, xbar + crit * se
    coverage = ((lo < mu) & (mu < hi)).mean()
    print(f"{label:>24}: coverage {coverage:.4f}")

Now the proportion intervals, where the standard method genuinely misbehaves:

import numpy as np
from scipy.stats import norm

def wald(k, n, z=1.96):
    p = k / n
    se = np.sqrt(p * (1 - p) / n)
    return p - z * se, p + z * se

def wilson(k, n, z=1.96):
    p = k / n
    d = 1 + z**2 / n
    centre = (p + z**2 / (2 * n)) / d
    half = z * np.sqrt(p * (1 - p) / n + z**2 / (4 * n**2)) / d
    return centre - half, centre + half

print(f"{'k/n':>10} {'Wald':>22} {'Wilson':>22}")
for k, n in [(0, 20), (1, 20), (5, 20), (10, 20), (19, 20), (20, 20)]:
    w = wald(k, n)
    s = wilson(k, n)
    print(f"{f'{k}/{n}':>10} ({w[0]:>7.4f}, {w[1]:>7.4f}) ({s[0]:>7.4f}, {s[1]:>7.4f})")

print("\nAt k=0 the Wald interval is [0, 0] -- certainty from no evidence.")

And the coverage comparison, which is the real argument:

import numpy as np

rng = np.random.default_rng(1)
n, trials, z = 30, 40_000, 1.96

print(f"{'true p':>8} {'Wald coverage':>16} {'Wilson coverage':>18}")
for p in (0.05, 0.1, 0.3, 0.5):
    k = rng.binomial(n, p, size=trials)
    phat = k / n

    se = np.sqrt(phat * (1 - phat) / n)
    wald_ok = ((phat - z*se < p) & (p < phat + z*se)).mean()

    d = 1 + z**2 / n
    centre = (phat + z**2 / (2*n)) / d
    half = z * np.sqrt(phat*(1-phat)/n + z**2/(4*n**2)) / d
    wilson_ok = ((centre - half < p) & (p < centre + half)).mean()

    print(f"{p:>8} {wald_ok:>16.4f} {wilson_ok:>18.4f}")

print("\nNominal coverage is 0.95. Wald falls short, badly so for small p.")

Your turn

1. n = 100, \bar x = 25, \sigma = 5 (known). Build a 95% CI.

2. How large must n be for a 95% CI to have width at most 1, if \sigma = 4?

3. A 95% CI for \mu is (10, 20). True or false: "$P(10 < \mu < 20) = 0.95$".

Solutions

1. \sigma is known, so use z:

$\text{SE} = \frac{5}{\sqrt{100}} = 0.5$

$25 \pm 1.96(0.5) = 25 \pm 0.98 = (24.02,\; 25.98)$

2. Width = 2 \times 1.96 \times \frac{\sigma}{\sqrt n} \le 1:

$\frac{2 \times 1.96 \times 4}{\sqrt n} \le 1 \implies \sqrt n \ge 15.68 \implies n \ge 245.9$

So n = 246.

This is the standard sample-size calculation, and it's why studies are planned around a target precision rather than a target result.

3. False, as a statement about this specific interval.

Once computed, (10, 20) either contains \mu or it doesn't — \mu is a fixed constant, not a random variable, so there is no probability to assign. The probability statement lived before the sample was drawn, when the endpoints were still random.

The correct version: "the procedure generating this interval captures \mu 95% of the time in repeated sampling."

This isn't pedantry with no consequences. It's exactly why a Bayesian credible interval is a different object: it does support the statement P(10 < \mu < 20) = 0.95, because Bayesian inference treats \mu as a random variable with a distribution. Same-looking numbers, different logic — §7.

Check yourself in code

Measure the actual coverage of t-based confidence intervals and confirm it lands near the nominal 95%.

Print exactly this:

coverage 0.9497
nominal 0.95
close enough: True

Use default_rng(0), \mu = 50, \sigma = 8, n = 12, 100000 trials. Round the coverage to 4 decimal places, and report close enough: True if it is within 0.005 of 0.95.

import numpy as np
from scipy.stats import t

rng = np.random.default_rng(0)
mu, sigma, n, trials = 50.0, 8.0, 12, 100_000

x = rng.normal(mu, sigma, size=(trials, n))
xbar = x.mean(axis=1)
se = x.std(axis=1, ddof=1) / np.sqrt(n)
crit = t.ppf(0.975, df=n - 1)

# Build the intervals, measure how often they contain mu, and report
# whether the coverage is within 0.005 of 0.95.

import numpy as np
from scipy.stats import t

rng = np.random.default_rng(0)
mu, sigma, n, trials = 50.0, 8.0, 12, 100_000

x = rng.normal(mu, sigma, size=(trials, n))
xbar = x.mean(axis=1)
se = x.std(axis=1, ddof=1) / np.sqrt(n)
crit = t.ppf(0.975, df=n - 1)

lo, hi = xbar - crit * se, xbar + crit * se
coverage = ((lo < mu) & (mu < hi)).mean()

print("coverage", round(coverage, 4))
print("nominal 0.95")
print("close enough:", bool(abs(coverage - 0.95) < 0.005))

A confidence interval is estimate ± critical value × standard error, and its confidence level describes the procedure, not any particular interval. Use t when \sigma is estimated, z when it's known or n is large. And treat the Wald interval for proportions with suspicion near the boundaries — Wilson is the better default.

Next: what "as good as possible, eventually" means precisely — the asymptotic criteria that separate estimators.

31. Confidence intervals