32. Consistency and asymptotic efficiency

Small-sample properties like unbiasedness are often unattainable, or attainable only at the cost of huge variance. So most of estimation theory is asymptotic: what happens as n \to \infty.

This lesson collects the asymptotic criteria and shows how they rank estimators.

Consistency

\hat\theta_n \xrightarrow{p} \theta

With enough data, the estimator converges to the truth. This is the minimum requirement — an inconsistent estimator doesn't improve no matter how much data you collect, which makes it close to useless.

The easiest sufficient condition, straight from the MSE decomposition:

\operatorname{Bias}(\hat\theta_n) \to 0 \;\text{ and }\; \operatorname{Var}(\hat\theta_n) \to 0 \implies \operatorname{MSE} \to 0 \implies \text{consistent}

(Convergence in L^2 implies convergence in probability — §3.)

Consistency and unbiasedness are independent properties:

Estimator	Unbiased?	Consistent?
\bar X for \mu	yes	yes
\hat\sigma^2 with divisor n	no	yes
X_1 for \mu	yes	no
\bar X + 1/n for \mu	no	yes

X_1 is the instructive case: perfectly unbiased at every n, and it never gets better, because it ignores the rest of the data. Unbiasedness on its own guarantees nothing.

Asymptotic normality

Most good estimators satisfy

\sqrt n\,(\hat\theta_n - \theta) \xrightarrow{d} N(0, v)

with v the asymptotic variance. The \sqrt n scaling is what keeps the limit from collapsing to a point — it's exactly the rate the CLT established.

This is what makes confidence intervals possible for estimators far more complicated than a sample mean: get v, and

\hat\theta \pm 1.96\sqrt{v/n}

Asymptotic efficiency

Among consistent, asymptotically Normal estimators, the one with the smallest asymptotic variance is asymptotically efficient. By Cramér–Rao (last lesson), the floor is

v \ge \frac{1}{I_1(\theta)}

and the MLE attains it:

\sqrt n\,(\hat\theta_{\text{MLE}} - \theta) \xrightarrow{d} N\!\left(0, \frac{1}{I_1(\theta)}\right)

That single fact is why maximum likelihood is the default method in statistics. It is asymptotically the best you can do.

Relative efficiency

To compare two estimators directly:

\text{ARE}(\hat\theta_1, \hat\theta_2) = \frac{v_2}{v_1}

An ARE of 2 means \hat\theta_1 achieves the same precision with half the data. That's the practical reading: efficiency is a sample-size multiplier.

The classic example. For a Normal sample, estimating \mu:

v_{\bar X} = \sigma^2, \qquad v_{\text{median}} = \frac{\pi\sigma^2}{2} \approx 1.571\sigma^2

\text{ARE}(\text{median}, \bar X) = \frac{2}{\pi} \approx 0.637

The median needs about 57% more data (1/0.637 = 1.57) to match the mean.

But that's for Normal data. For heavy-tailed data the ranking reverses completely — for a Cauchy, the sample mean is inconsistent (§3) while the median is perfectly well behaved. Efficiency is always relative to an assumed model, and "optimal under the Normal" can mean "catastrophic under contamination".

That trade-off is the subject of robust statistics: give up some efficiency at the assumed model in exchange for not failing when the model is wrong.

The asymptotic caveat

These are all limiting statements, and n \to \infty never actually happens.

• The MLE can be badly biased at small n (the Normal variance again).
• Convergence to Normality can be slow for skewed distributions.
• Asymptotic standard errors can be too optimistic in small samples.

Asymptotic theory tells you what happens eventually. Whether your n counts as "eventually" is an empirical question — and simulation, as below, is the usual way to answer it.

Worked example

Compare \bar X and the median for estimating the centre of a Normal, then for a contaminated Normal.

Clean Normal. As above, $\text{ARE}(\text{median}, \bar X) = 2/\pi \approx 0.64$ — the mean wins clearly.

Contaminated: 95% from N(0,1), 5% from N(0, 10^2). The variance of a single observation is now

0.95(1) + 0.05(100) = 5.95

so v_{\bar X} = 5.95. The median, however, barely notices — the contamination sits in the tails and the middle of the distribution is almost unchanged, giving v_{\text{median}} \approx 1.7.

\text{ARE}(\text{median}, \bar X) \approx \frac{5.95}{1.7} \approx 3.5

The ranking has completely flipped. Five percent contamination is enough to make the median over three times more efficient than the mean.

This is the practical argument for robust estimators, and it's why §0 recommended the median for skewed data. Optimality is conditional on the model being right.

Doing it in Python

The ARE of the median under a clean Normal, measured directly:

import numpy as np

rng = np.random.default_rng(0)
n, trials = 101, 200_000

x = rng.standard_normal((trials, n))
mean_est = x.mean(axis=1)
med_est = np.median(x, axis=1)

print(f"variance of mean  : {mean_est.var():.6f}   (theory {1/n:.6f})")
print(f"variance of median: {med_est.var():.6f}   (theory {np.pi/(2*n):.6f})")
print(f"\nARE(median, mean) = {mean_est.var()/med_est.var():.4f}   (theory {2/np.pi:.4f})")
print("-> under a clean Normal, the mean is clearly better")

Now contaminate the data and watch the ranking invert:

import numpy as np

rng = np.random.default_rng(1)
n, trials = 101, 200_000

clean = rng.standard_normal((trials, n))
outlier = rng.standard_normal((trials, n)) * 10
mask = rng.random((trials, n)) < 0.05
x = np.where(mask, outlier, clean)          # 5% contamination

mean_est = x.mean(axis=1)
med_est = np.median(x, axis=1)

print(f"variance of mean  : {mean_est.var():.6f}")
print(f"variance of median: {med_est.var():.6f}")
print(f"\nARE(median, mean) = {mean_est.var()/med_est.var():.4f}")
print("-> with 5% contamination, the median is now several times better")

Consistency, or its absence:

import numpy as np

rng = np.random.default_rng(2)
mu, trials = 5.0, 100_000

print(f"{'n':>8} {'sd of Xbar':>14} {'sd of X_1':>12}")
for n in (10, 100, 1_000, 10_000):
    x = rng.normal(mu, 2.0, size=(trials, n))
    print(f"{n:>8} {x.mean(axis=1).std():>14.5f} {x[:, 0].std():>12.5f}")

print("\nXbar tightens like 1/sqrt(n) -- consistent.")
print("X_1 never moves -- unbiased at every n, and never converges.")

And how quickly the MLE reaches its asymptotic promise:

import numpy as np

rng = np.random.default_rng(3)
lam, trials = 2.0, 100_000
# MLE of an Exponential rate is 1/xbar; asymptotic variance is lambda^2 / n
print(f"{'n':>8} {'MLE mean':>12} {'MLE var':>12} {'asymptotic':>12}")
for n in (5, 20, 100, 1_000):
    x = rng.exponential(1 / lam, size=(trials, n))
    mle = 1 / x.mean(axis=1)
    print(f"{n:>8} {mle.mean():>12.4f} {mle.var():>12.6f} {lam**2/n:>12.6f}")

print(f"\nTrue lambda = {lam}. At n=5 the MLE is biased high and its variance")
print("exceeds the asymptotic value. Both problems fade as n grows.")

Your turn

1. Is \hat\theta_n = \bar X + \frac{1}{n} consistent for \mu? Unbiased?

2. Estimator A has asymptotic variance 2/n, B has 3/n. What's the ARE, and what does it mean practically?

3. Why might you use the median even though it's less efficient for Normal data?

Solutions

1. Biased, at every finite n:

E[\hat\theta_n] = \mu + \frac{1}{n} \ne \mu

Consistent, though. The bias \to 0 and the variance \sigma^2/n \to 0, so MSE \to 0.

More directly: \hat\theta_n = \bar X + \frac1n, where $\bar X \xrightarrow{p} \mu$ and \frac1n \to 0. Slutsky's theorem (§3) gives $\hat\theta_n \xrightarrow{p} \mu$.

A neat illustration that the two properties are genuinely independent — this estimator fails one and satisfies the other, and asymptotically it's just as good as \bar X.

2.

\text{ARE}(A, B) = \frac{v_B}{v_A} = \frac{3}{2} = 1.5

A is 1.5 times as efficient. Practically: A achieves with n observations what B needs 1.5n observations to match. If B needs 300 samples, A needs 200.

Efficiency is best read as a sample-size multiplier, which converts directly into time and cost.

3. Several reasons, and all of them are about the model being wrong:

• Robustness to outliers. The mean has breakdown point 0 — a single arbitrarily large value drags it anywhere. The median's breakdown point is 50%: it survives until half the data is corrupted.

• Heavy tails. For a Cauchy, the mean is inconsistent (§3) while the median converges normally. Here the efficiency comparison isn't close — one estimator works and the other doesn't.

• Skewness. For skewed data the median is often the more meaningful summary anyway, as §0's salary example showed. The "centre" you want isn't always the mean.

The cost is 36% efficiency under an exactly-Normal model. Real data is rarely exactly Normal, so that insurance premium is usually worth paying.

Check yourself in code

Measure the asymptotic relative efficiency of the median against the mean for Normal data, and confirm it matches the theoretical 2/\pi.

Print exactly this:

var mean 0.009852
var median 0.015472
ARE 0.6368
matches 2/pi: True

Use default_rng(0), n = 101, 200000 trials of standard Normal data. Round the variances to 6 decimal places and the ARE to 4. Report matches 2/pi: True if the ARE is within 0.02 of 2/\pi.

import numpy as np

rng = np.random.default_rng(0)
n, trials = 101, 200_000

x = rng.standard_normal((trials, n))
mean_est = x.mean(axis=1)
med_est = np.median(x, axis=1)

print("var mean", round(mean_est.var(), 6))

# Print the median's variance, the ARE (var mean / var median),
# and whether it is within 0.02 of 2/pi.

import numpy as np

rng = np.random.default_rng(0)
n, trials = 101, 200_000

x = rng.standard_normal((trials, n))
mean_est = x.mean(axis=1)
med_est = np.median(x, axis=1)

print("var mean", round(mean_est.var(), 6))
print("var median", round(med_est.var(), 6))

are = mean_est.var() / med_est.var()
print("ARE", round(are, 4))
print("matches 2/pi:", bool(abs(are - 2 / np.pi) < 0.02))

---

Consistency is the minimum bar: converge to the truth with enough data. Asymptotic normality gives you standard errors and intervals. Asymptotic efficiency ranks estimators by their limiting variance, with the MLE attaining the Cramér–Rao floor. But every efficiency claim is conditional on the assumed model — and under contamination the rankings can reverse completely.

That closes §4. Next: instead of estimating a parameter, deciding between two claims about it.