22. Modes of convergence
For ordinary numbers, "a_n \to a" means one thing. For random variables there are several genuinely different things it could mean, and the limit theorems ahead each use a specific one.
Getting these straight is what lets you read "the sample mean converges to \mu" and know exactly what is being claimed.
Four modes
Throughout, X_n is a sequence of random variables and X is the limit.
1. In distribution (X_n \xrightarrow{d} X)
F_{X_n}(x) \to F_X(x) \quad \text{at every } x \text{ where } F_X \text{ is continuous}
Only the distributions converge. The variables themselves need not settle down at all — they needn't even be defined on the same sample space.
This is the weakest mode, and it's exactly what the Central Limit Theorem delivers.
The "at continuity points" clause is a genuine technicality, not decoration. Let X_n be the constant 1/n. Then F_{X_n} jumps at 1/n, while the limit X = 0 has its jump at 0. At the point x = 0 we have F_{X_n}(0) = 0 for all n but F_X(0) = 1 — convergence fails exactly at the discontinuity, and nowhere else. Without the clause, this obvious convergence wouldn't count.
2. In probability (X_n \xrightarrow{p} X)
P\big(|X_n - X| > \varepsilon\big) \to 0 \quad \text{for every } \varepsilon > 0
The probability of being far from the limit vanishes. X_n can still deviate occasionally — but such deviations get rarer and rarer.
This is what the Weak Law of Large Numbers gives.
3. Almost surely (X_n \xrightarrow{a.s.} X)
P\Big(\lim_{n\to\infty} X_n = X\Big) = 1
Look at an individual infinite sequence of outcomes. With probability 1, that sequence converges in the ordinary calculus sense.
This is stronger, and the difference is subtle but real. Convergence in probability says "at each fixed large n, being far off is unlikely". Almost-sure convergence says "eventually you stop being far off, forever".
A sequence can converge in probability while, along the way, deviating infinitely often. The Strong Law of Large Numbers rules that out.
4. In L^p (X_n \xrightarrow{L^p} X)
E\big[|X_n - X|^p\big] \to 0
The case p = 2 (mean square convergence) is the common one, and it's often the easiest to verify — computing E[(X_n - X)^2] is usually just a variance calculation.
How they relate
L^p \;\Longrightarrow\; \text{in probability} \;\Longrightarrow\; \text{in distribution}
\text{almost surely} \;\Longrightarrow\; \text{in probability} \;\Longrightarrow\; \text{in distribution}
Convergence in distribution is the weakest; almost-sure and L^p are both stronger than convergence in probability but neither implies the other.
One partial converse is worth knowing:
X_n \xrightarrow{d} c \;\text{ (a constant)} \implies X_n \xrightarrow{p} c
Converging in distribution to a constant is enough to give convergence in probability. This is used constantly — it's how Slutsky's theorem gets applied two lessons from now.
Why the distinctions matter
Convergence in distribution is not about the variables. Let X \sim N(0,1) and set X_n = -X for every n. Then X_n has exactly the same distribution as X (the standard Normal is symmetric), so trivially X_n \xrightarrow{d} X.
But |X_n - X| = |{-2X}| = 2|X|, which is nowhere near 0. There's no convergence in probability at all. The distributions match; the variables don't.
Convergence in probability doesn't stop infinite deviations. The standard example is the "typewriter" sequence. On [0,1] with uniform probability, define indicator variables on shrinking intervals that sweep across the unit interval repeatedly: [0,1], then [0,\frac12], [\frac12,1], then [0,\frac13],[\frac13,\frac23],[\frac23,1], and so on.
Each X_n is 1 on its interval and 0 elsewhere. The interval widths \to 0, so P(X_n = 1) \to 0 and X_n \xrightarrow{p} 0.
But every point of [0,1] is covered by infinitely many of these intervals, so for every outcome the sequence X_n hits 1 infinitely often and never settles. It converges in probability but not almost surely.
Worked example
X_n \sim \text{Uniform}(0, 1/n). Show X_n \xrightarrow{p} 0, and in L^2.
In probability. Fix any \varepsilon > 0. Once n is large enough that 1/n < \varepsilon, the variable can't exceed \varepsilon at all:
P(|X_n| > \varepsilon) = P(X_n > \varepsilon) = \begin{cases} 1 - n\varepsilon & \varepsilon < 1/n \\ 0 & \varepsilon \ge 1/n\end{cases}
Either way this \to 0 as n \to \infty. ✓
In L^2. For a Uniform(0, b), E[X^2] = b^2/3, so
E[X_n^2] = \frac{1}{3n^2} \to 0 \quad\checkmark
In distribution. F_{X_n}(x) \to 1 for every x > 0 and = 0 for x < 0 — the CDF of the constant 0. ✓ (And note we can't check x = 0 itself: that's the discontinuity point the definition excuses.)
All modes hold here, which is typical of well-behaved sequences. The distinctions bite only in the constructed examples above — but those examples are exactly why the theorems are stated with the mode named.
Doing it in Python
Convergence in probability, watched directly:
import numpy as np
rng = np.random.default_rng(0)
eps = 0.05
print(f"{'n':>8} {'P(|X_n| > 0.05)':>18} {'E[X_n^2]':>12}")
for n in (1, 2, 5, 10, 50, 200):
X = rng.uniform(0, 1 / n, size=200_000)
print(f"{n:>8} {(np.abs(X) > eps).mean():>18.4f} {(X**2).mean():>12.6f}")
Both columns march to zero — the first is convergence in probability, the second is L^2.
Now the counterexample that separates distribution from probability:
import numpy as np
rng = np.random.default_rng(1)
X = rng.standard_normal(200_000)
Xn = -X # same distribution, opposite values
print("same distribution?")
print(" mean:", round(X.mean(), 4), "vs", round(Xn.mean(), 4))
print(" sd :", round(X.std(), 4), "vs", round(Xn.std(), 4))
for q in (0.1, 0.5, 0.9):
print(f" q={q}: {np.quantile(X, q):+.4f} vs {np.quantile(Xn, q):+.4f}")
print("\nbut are the VALUES close?")
print(" mean |Xn - X| =", round(np.abs(Xn - X).mean(), 4))
print(" P(|Xn - X| > 0.5) =", round((np.abs(Xn - X) > 0.5).mean(), 4))
print(" -> converges in distribution, NOT in probability")
And the typewriter sequence, showing probability-but-not-almost-surely:
import numpy as np
# Interval k of block m covers [k/m, (k+1)/m); blocks m = 1, 2, 3, ...
def intervals(n_terms):
out, m, k = [], 1, 0
while len(out) < n_terms:
out.append((k / m, (k + 1) / m))
k += 1
if k == m:
m, k = m + 1, 0
return out
terms = intervals(60)
omega = 0.4213 # one fixed outcome
hits = [1 if lo <= omega < hi else 0 for lo, hi in terms]
widths = [hi - lo for lo, hi in terms]
print("P(X_n = 1) = interval width, which shrinks:")
print(" ", [round(w, 3) for w in widths[:6]], "...", round(widths[-1], 3))
print("\nbut for the single outcome w = 0.4213 the sequence keeps spiking:")
print(" ", "".join(str(h) for h in hits))
print(f"\n spikes in the last 30 terms: {sum(hits[30:])}")
print(" -> X_n -> 0 in probability, but never settles for this outcome")
Your turn
1. X_n = 1/n (constant, non-random). Which modes hold?
2. X_n \sim N(0, 1/n). Show X_n \xrightarrow{p} 0.
3. Does convergence in distribution to a constant imply convergence in probability?
Solutions
1. All of them. A non-random sequence converging in the ordinary sense converges in every probabilistic mode:
- Almost surely: the single sequence 1/n \to 0, with probability 1. ✓
- In probability: P(|1/n - 0| > \varepsilon) = 0 once n > 1/\varepsilon. ✓
- In L^p: E[|1/n|^p] = 1/n^p \to 0. ✓
- In distribution: follows from any of the above. ✓
Deterministic sequences are the degenerate case where all four coincide, which is why they're a poor guide to the distinctions.
2. Use Chebyshev's inequality, $P(|X - \mu| \ge \varepsilon) \le \sigma^2/\varepsilon^2$. Here \mu = 0 and \sigma^2 = 1/n:
P(|X_n| > \varepsilon) \le \frac{1/n}{\varepsilon^2} = \frac{1}{n\varepsilon^2} \to 0
for every fixed \varepsilon > 0. ✓
Convergence in L^2 is even quicker: $E[X_n^2] = \operatorname{Var}(X_n) = 1/n \to 0$, and L^2 convergence implies convergence in probability. That's the usual shortcut — verify the mean-square statement, get the probability one free.
3. Yes — this is the one converse that holds.
Intuition: convergence in distribution says the CDFs approach a step function at c. A step at c means all the probability piles up at c, leaving none outside any interval around it. Formally, for X_n \xrightarrow{d} c:
P(|X_n - c| > \varepsilon) = P(X_n < c - \varepsilon) + P(X_n > c + \varepsilon) \to 0 + 0 = 0
since both c \pm \varepsilon are continuity points of the limiting CDF.
The result fails for non-constant limits — that's the X_n = -X example, where the limit is a genuine distribution rather than a point.
Check yourself in code
For X_n \sim \text{Uniform}(0, 1/n), confirm convergence both in probability and in L^2 by showing both quantities shrink as n grows.
Print exactly this:
n=1 P=0.9505 L2=0.332381
n=10 P=0.5002 L2=0.003341
n=100 P=0.0 L2=3.3e-05
converges: True
Use default_rng(0) and 200000 draws per n. Print P(|X_n| > 0.05) rounded
to 4 decimal places and E[X_n^2] rounded to 6. Report converges: True if
both quantities strictly decrease across the three values of n.
import numpy as np
rng = np.random.default_rng(0)
eps = 0.05
probs, l2s = [], []
for n in (1, 10, 100):
X = rng.uniform(0, 1 / n, size=200_000)
p = round((np.abs(X) > eps).mean(), 4)
m = round((X**2).mean(), 6)
probs.append(p)
l2s.append(m)
print(f"n={n} P={p} L2={m}")
# Report whether both sequences are strictly decreasing.
import numpy as np
rng = np.random.default_rng(0)
eps = 0.05
probs, l2s = [], []
for n in (1, 10, 100):
X = rng.uniform(0, 1 / n, size=200_000)
p = round((np.abs(X) > eps).mean(), 4)
m = round((X**2).mean(), 6)
probs.append(p)
l2s.append(m)
print(f"n={n} P={p} L2={m}")
decreasing = all(a > b for a, b in zip(probs, probs[1:])) and \
all(a > b for a, b in zip(l2s, l2s[1:]))
print("converges:", decreasing)
Four modes, in increasing strength: distribution, probability, then almost-sure and L^p (which are incomparable with each other). Convergence in distribution constrains only the CDFs, not the variables. Convergence in probability allows deviations that recur forever. And converging in distribution to a constant is the one case where the weakest mode upgrades itself.
Next: the theorem that says sample means converge at all — and which of these modes it delivers.