56. Probability measures

With the events pinned down, we can say precisely what a probability is: a measure of total mass 1.

The striking part is how little is special about probability. It's an instance of a general theory that also covers length, area, volume and mass — and recognising that is what lets one set of theorems serve all of them.

Measures

A measure on (\Omega, \mathcal F) is a function $\mu : \mathcal F \to [0, \infty]$ with:

1. \mu(\varnothing) = 0
2. Countable additivity: for pairwise disjoint A_1, A_2, \dots,

\mu\left(\bigcup_{i=1}^\infty A_i\right) = \sum_{i=1}^\infty \mu(A_i)

That's all. The triple (\Omega, \mathcal F, \mu) is a measure space.

A probability measure is a measure with \mu(\Omega) = 1. Written P, and (\Omega, \mathcal F, P) is a probability space.

Compare with Kolmogorov's axioms from §0:

§0 axiom	Measure-theoretic version
P(A) \ge 0	\mu maps into [0,\infty]
P(\Omega) = 1	the normalisation \mu(\Omega) = 1
Additivity for disjoint events	countable additivity

They're the same axioms. §0 gave them informally; here they're the definition of a standard mathematical object, which means every theorem about measures applies immediately to probability.

Familiar measures

Lebesgue measure \lambda on \mathbb R: the generalisation of length, with \lambda([a,b]) = b - a. Not a probability measure — \lambda(\mathbb R) = \infty — but restrict it to [0,1] and it becomes the uniform distribution.

Counting measure: \mu(A) = |A|. Normalise on a finite set and you get the classical equally-likely probability from §0.

Dirac measure \delta_x: mass 1 at the point x, zero elsewhere. The distribution of a constant.

The distribution of a random variable. Given X on $(\Omega, \mathcal F, P)$, define a measure on \mathbb R by

P_X(B) = P(X^{-1}(B)) = P(X \in B)

This pushforward measure is what we've been calling "the distribution of X" all along. Everything in §1 — PMFs, PDFs, CDFs — is a way of describing P_X.

Properties that follow

Everything §0 derived, now as theorems about measures:

P(A^c) = 1 - P(A), \qquad A \subseteq B \implies P(A) \le P(B)

P(A \cup B) = P(A) + P(B) - P(A \cap B)

Two more that need countable additivity:

Continuity from below. If A_1 \subseteq A_2 \subseteq \cdots then

P\left(\bigcup_n A_n\right) = \lim_{n\to\infty} P(A_n)

Continuity from above. If A_1 \supseteq A_2 \supseteq \cdots then P\left(\bigcap_n A_n\right) = \lim_n P(A_n).

These are what make limit theorems possible. Every statement of the form "the probability of eventual convergence" relies on being able to pass a limit through P, and continuity is exactly that permission.

Countable subadditivity (no disjointness needed):

P\left(\bigcup_n A_n\right) \le \sum_n P(A_n)

This is the union bound, and despite being crude it is one of the most-used inequalities in all of probability — it's how you bound the chance that any of many bad events occurs, which is exactly the multiple-comparisons calculation from §5.

Null sets and "almost surely"

A set N is null if P(N) = 0.

A property holds almost surely (a.s.) if it holds outside some null set. This is the phrase that appears throughout §3's convergence modes, and now it has a precise meaning.

The crucial point, first met in §1: probability zero does not mean impossible.

Pick a uniform point in [0,1]. The probability of getting exactly 0.5 is 0 — yet some number is picked, and whatever it was had probability 0 beforehand. Every individual outcome is a null event; the rationals as a whole are a null set despite being dense and infinite.

The reason is countable additivity: a countable union of null sets is null. The rationals are countable, so \lambda(\mathbb Q) = 0. The irrationals, being uncountable, carry all the mass.

Lebesgue's construction

How is a measure actually built? You cannot specify \lambda(A) for every Borel set directly.

The standard route:

1. Define \lambda on intervals: \lambda([a,b]) = b - a.
2. Extend to an outer measure by covering: $\lambda^*(A) = \inf{\sum (b_i - a_i) : A \subseteq \bigcup (a_i,b_i)}$.
3. Restrict to the sets where the outer measure behaves additively (Carathéodory's criterion).

Carathéodory's extension theorem guarantees the result is a genuine measure and that it's the unique one agreeing with the interval lengths. That uniqueness is what lets you define a distribution by its CDF alone — specify P((-\infty, x]) for every x and the whole measure is determined.

Worked example

Show that P(\mathbb Q \cap [0,1]) = 0 under the uniform distribution.

The rationals are countable, so enumerate them: q_1, q_2, q_3, \dots

Fix \varepsilon > 0 and cover q_i with an interval of width \varepsilon/2^i:

\mathbb Q \cap [0,1] \subseteq \bigcup_{i=1}^\infty \left(q_i - \frac{\varepsilon}{2^{i+1}},\; q_i + \frac{\varepsilon}{2^{i+1}}\right)

By countable subadditivity:

P(\mathbb Q \cap [0,1]) \le \sum_{i=1}^\infty \frac{\varepsilon}{2^i} = \varepsilon

This holds for every \varepsilon > 0, so the probability is 0. ∎

A dense set of measure zero. Between any two reals there's a rational, yet the rationals occupy no length at all. A uniform random draw is irrational with probability 1.

Note where countability was essential: the trick of assigning shrinking widths \varepsilon/2^i needs an enumeration. The irrationals cannot be covered this way, and they carry the entire measure.

Doing it in Python

Countable additivity and continuity, on nested sequences:

import numpy as np

# Uniform on [0,1]. A_n = [0, 1 - 1/n] increases to [0, 1).
print("continuity from below: A_n = [0, 1 - 1/n]")
for n in (2, 10, 100, 10_000, 1_000_000):
    print(f"  n = {n:>9}: P(A_n) = {1 - 1/n:.8f}")
print("  limit = 1.0 = P(union) \n")

# B_n = [0, 1/n] decreases to {0}, a null set.
print("continuity from above: B_n = [0, 1/n]")
for n in (2, 10, 100, 10_000, 1_000_000):
    print(f"  n = {n:>9}: P(B_n) = {1/n:.8f}")
print("  limit = 0 = P(intersection) = P({0})")

The rationals have measure zero — the covering argument, made concrete:

from fractions import Fraction

# Enumerate rationals in [0,1] by denominator (a genuine enumeration)
def rationals(limit):
    seen, out = set(), []
    d = 1
    while len(out) < limit:
        for n in range(d + 1):
            q = Fraction(n, d)
            if q not in seen:
                seen.add(q)
                out.append(q)
        d += 1
    return out[:limit]

qs = rationals(2000)
print("first few rationals:", [str(q) for q in qs[:8]])

for eps in (1.0, 0.1, 0.001):
    # Widths eps/2^(i+1) form a geometric series summing to at most eps.
    # (Only the first 60 terms matter numerically; the rest underflow.)
    total = sum(eps / 2 ** (i + 1) for i in range(60))
    print(f"  epsilon = {eps:>6}: total covering width <= {total:.6f}")

print("\nThe cover can be made as small as we like, for ANY epsilon.")
print("So the rationals have measure exactly 0 -- despite being dense.")

And empirically: a uniform draw is never rational.

import numpy as np
from fractions import Fraction

rng = np.random.default_rng(0)
draws = rng.uniform(0, 1, 200_000)

# How many land on a rational with a small denominator?
hits = 0
for d in range(1, 101):
    for n in range(d + 1):
        hits += (draws == n / d).sum()

print(f"draws landing exactly on a rational with denominator <= 100: {hits}")
print("(Floating point makes this a coarse test, but the point stands:")
print(" P(rational) = 0, so it essentially never happens.)")

The union bound — crude but everywhere:

import numpy as np

rng = np.random.default_rng(1)

# m independent events, each with probability p. Union bound says
# P(at least one) <= m*p, with equality only when they are disjoint.
p, trials = 0.01, 200_000
print(f"{'m':>6} {'simulated P(any)':>18} {'union bound m*p':>18} {'exact 1-(1-p)^m':>18}")
for m in (1, 5, 20, 50, 100):
    events = rng.random((trials, m)) < p
    print(f"{m:>6} {events.any(axis=1).mean():>18.4f} {min(m*p, 1):>18.4f} "
          f"{1 - (1-p)**m:>18.4f}")

print("\nThe bound is tight when p is small and loose when it isn't --")
print("which is exactly when Bonferroni correction is reasonable (§5).")

Your turn

1. Is Lebesgue measure on \mathbb R a probability measure?

2. Give an event with probability 0 that is not impossible.

3. If P(A_n) = 0 for every n, what is P(\bigcup_n A_n)?

Solutions

1. No. A probability measure requires \mu(\Omega) = 1, but

\lambda(\mathbb R) = \infty

Lebesgue measure is \sigma-finite (the line is a countable union of finite-measure intervals) but not finite, so it can't be normalised to 1.

Restricted to [0,1], however, it is a probability measure — that's exactly the uniform distribution. And on any bounded interval [a,b], dividing by b - a normalises it.

2. Draw X uniformly from [0,1] and consider the event \{X = 0.5\}.

P(X = 0.5) = 0

yet 0.5 is a perfectly possible outcome — it's in the sample space, and nothing excludes it. In fact whatever value is drawn had probability 0 beforehand, so "probability 0" happens on every single draw.

Other examples: \{X \in \mathbb Q\} (the rationals, as shown above), or "a Brownian path is differentiable at some fixed time t".

The distinction matters in practice. Bayesian priors assigning exactly zero to a region make it permanently unreachable (§7's Cromwell's rule), whereas probability-zero events under a continuous distribution are simply the norm.

3. Zero. By countable subadditivity:

P\left(\bigcup_n A_n\right) \le \sum_n P(A_n) = \sum_n 0 = 0

A countable union of null sets is null. This is used constantly: it's why "almost surely" statements can be combined — if each of countably many properties holds a.s., then all of them hold simultaneously a.s.

The countability is essential. [0,1] is an uncountable union of singletons, each of measure 0, and it has measure 1. That's precisely the gap the Vitali set exploited in the last lesson.

Check yourself in code

Demonstrate continuity of measure from below and above on nested sequences, and confirm the union bound.

Print exactly this:

P(A_n) at n=1000000 0.999999
continuity from below: True
P(B_n) at n=1000000 1e-06
continuity from above: True
union bound holds: True

Take A_n = [0, 1 - 1/n] and B_n = [0, 1/n] under the uniform distribution on [0,1]. For the union bound, check 1 - (1-p)^m \le mp for p = 0.01 and m \in \{1, 5, 20, 50, 100\}.

n = 1_000_000

print("P(A_n) at n=1000000", round(1 - 1/n, 6))
print("continuity from below:", bool(abs((1 - 1/n) - 1) < 1e-5))

# Print P(B_n) = 1/n, whether it approaches 0, and whether the union bound
# 1 - (1-p)**m <= m*p holds for p = 0.01 and m in (1, 5, 20, 50, 100).

n = 1_000_000

print("P(A_n) at n=1000000", round(1 - 1/n, 6))
print("continuity from below:", bool(abs((1 - 1/n) - 1) < 1e-5))
print("P(B_n) at n=1000000", 1/n)
print("continuity from above:", bool(1/n < 1e-5))

p = 0.01
print("union bound holds:", all(1 - (1 - p)**m <= m * p + 1e-12
                                for m in (1, 5, 20, 50, 100)))

---

A probability measure is just a measure normalised to total mass 1, so Kolmogorov's axioms turn out to be an instance of a much more general theory. Countable additivity buys continuity along nested sequences — which is what makes limit theorems statable — and subadditivity gives the union bound. Null sets formalise "almost surely", and remind us that probability zero and impossibility are different things.

Next: how to integrate against a measure, and why it's better than the integral you already know.