59. Radon–Nikodym and conditional expectation
Two final ideas, and they turn out to be the same idea.
The first explains what a probability density actually is. The second gives conditional expectation — used informally since §0 and relied on heavily in §8 — its proper definition.
Absolute continuity
A measure \nu is absolutely continuous with respect to \mu, written \nu \ll \mu, if
\mu(A) = 0 \implies \nu(A) = 0
Whatever \mu considers negligible, \nu must too. \nu cannot put mass where \mu sees nothing.
The opposite extreme is mutual singularity (\nu \perp \mu): the two live on disjoint sets, each assigning full mass to a set the other calls null.
The Radon–Nikodym theorem
If \nu \ll \mu and both are \sigma-finite, there exists an essentially unique non-negative measurable f with
\nu(A) = \int_A f\,d\mu \quad \text{for every measurable } A
f is the Radon–Nikodym derivative, written
f = \frac{d\nu}{d\mu}
The notation is deliberate: it behaves like a derivative, with a chain rule \frac{d\nu}{d\mu} = \frac{d\nu}{d\rho}\cdot\frac{d\rho}{d\mu}, and it's unique up to null sets.
This is what a density is
A continuous random variable's PDF is exactly the Radon–Nikodym derivative of its distribution with respect to Lebesgue measure:
f_X = \frac{dP_X}{d\lambda}
"X has a density" simply means P_X \ll \lambda — the distribution puts no mass on Lebesgue-null sets.
That single sentence explains several things at once:
- Why discrete variables have no density. A discrete P_X puts mass 1 on a countable set, which has Lebesgue measure 0. So P_X \not\ll \lambda, and no density exists. (Discrete variables do have a density with respect to counting measure — and that density is the PMF. The dichotomy from §1 was never fundamental; it was a choice of reference measure.)
- Why densities aren't unique. Radon–Nikodym derivatives are defined only up to null sets, so changing f at finitely many points gives the same distribution.
- Why mixed distributions are fine. The insurance example from two lessons ago is absolutely continuous with respect to (Lebesgue + a Dirac mass), and has a density with respect to that.
The likelihood ratio in §5's Neyman–Pearson lemma is also a Radon–Nikodym derivative, \frac{dP_1}{dP_0} — which is why it's the natural object to compare two hypotheses with.
Conditional expectation, properly
§0 defined P(A \mid B) = P(A \cap B)/P(B), which fails when P(B) = 0. But we routinely condition on continuous variables, where every value has probability zero. What does E[Y \mid X = x] mean?
The measure-theoretic answer changes the question. Instead of conditioning on a single event, condition on a whole \sigma-algebra $\mathcal G \subseteq \mathcal F$ — an information set (§9, lesson 1).
Definition. E[Y \mid \mathcal G] is any random variable Z such that:
- Z is \mathcal G-measurable — computable from the information in \mathcal G.
- For every A \in \mathcal G:
\int_A Z\,dP = \int_A Y\,dP
Such a Z exists and is unique up to null sets — and its existence is a corollary of Radon–Nikodym. Define \nu(A) = \int_A Y\,dP on \mathcal G; then \nu \ll P, and Z = d\nu/dP is the conditional expectation.
Reading the definition
Condition 1 says: you may only use information you actually have.
Condition 2 says: Z has the same average as Y over every set you can distinguish. It doesn't match Y pointwise — that would need full information — but it gets every observable average right.
Note that E[Y \mid \mathcal G] is a random variable, not a number. Its value depends on what \mathcal G reveals.
The projection view
In L^2, conditional expectation is orthogonal projection of Y onto the space of \mathcal G-measurable functions.
E[Y \mid \mathcal G] = \text{the } \mathcal G\text{-measurable } Z \text{ minimising } E[(Y - Z)^2]
It is the best prediction of Y using only the available information. That's the same optimality as least squares (§6), and it's why the regression function E[Y \mid X] is what regression is trying to estimate.
Properties
E\big[E[Y \mid \mathcal G]\big] = E[Y] \qquad \textbf{(tower / law of total expectation)}
\mathcal G_1 \subseteq \mathcal G_2 \implies E\big[E[Y \mid \mathcal G_2] \mid \mathcal G_1\big] = E[Y \mid \mathcal G_1]
Less information wins — averaging a refined estimate down to a coarser information set gives the coarse estimate directly.
E[XY \mid \mathcal G] = X\,E[Y \mid \mathcal G] \quad \text{for } \mathcal G\text{-measurable } X
Known quantities factor out.
\mathcal G = \{\varnothing, \Omega\} \implies E[Y \mid \mathcal G] = E[Y]
No information gives the unconditional mean — a constant, as §9 lesson 1's exercise predicted.
The martingale condition from §8, E[X_{n+1} \mid \mathcal F_n] = X_n, is now fully precise: it says the projection of tomorrow onto today's information is today's value.
Worked example
Roll a die. Let Y be the outcome and \mathcal G the parity information. Find E[Y \mid \mathcal G].
\mathcal G = \{\varnothing, \{1,3,5\}, \{2,4,6\}, \Omega\}.
A \mathcal G-measurable variable must be constant on each of \{1,3,5\} and \{2,4,6\} — those are the finest distinctions \mathcal G can make.
Match averages over each:
Z = \frac{1+3+5}{3} = 3 \text{ on odds}, \qquad Z = \frac{2+4+6}{3} = 4 \text{ on evens}
So E[Y \mid \mathcal G] = 3 if the roll is odd, 4 if even.
Check the tower property:
E[Z] = \tfrac12(3) + \tfrac12(4) = 3.5 = E[Y] \quad\checkmark
Check condition 2 on A = \{2,4,6\}:
\int_A Z\,dP = 4 \times \tfrac12 = 2, \qquad \int_A Y\,dP = \tfrac{2+4+6}{6} = 2 \quad\checkmark
Note the loss of information. Z takes only two values where Y took six. It is the best possible guess given parity alone — and
E[(Y - Z)^2] = \frac{4 + 0 + 4 + 4 + 0 + 4}{6} = \frac{16}{6} \approx 2.667
is the irreducible error, the variance parity cannot remove. Compare with \operatorname{Var}(Y) = 35/12 \approx 2.917: knowing the parity removes only about 9% of the variance. It's genuine information, but not much of it.
Doing it in Python
Conditional expectation as an average over each information block:
import numpy as np
omega = np.array([1, 2, 3, 4, 5, 6])
Y = omega.astype(float)
def conditional_expectation(Y, blocks):
"""E[Y | G], where `blocks` are the finest sets G can distinguish."""
Z = np.zeros_like(Y)
for block in blocks:
idx = [np.where(omega == w)[0][0] for w in block]
Z[idx] = Y[idx].mean()
return Z
parity = [{1, 3, 5}, {2, 4, 6}]
Z = conditional_expectation(Y, parity)
print(f"{'outcome':>9} {'Y':>5} {'E[Y|parity]':>13}")
for w, y, z in zip(omega, Y, Z):
print(f"{w:>9} {y:>5.0f} {z:>13.4f}")
print(f"\nE[Z] = {Z.mean():.4f} E[Y] = {Y.mean():.4f} (tower property)")
print(f"E[(Y - Z)^2] = {((Y - Z)**2).mean():.4f} <- information parity cannot recover")
Finer information gives a better projection:
import numpy as np
omega = np.array([1, 2, 3, 4, 5, 6])
Y = omega.astype(float)
def cond_exp(blocks):
Z = np.zeros_like(Y)
for block in blocks:
idx = [np.where(omega == w)[0][0] for w in block]
Z[idx] = Y[idx].mean()
return Z
information = {
"nothing": [{1, 2, 3, 4, 5, 6}],
"parity": [{1, 3, 5}, {2, 4, 6}],
"high/low": [{1, 2, 3}, {4, 5, 6}],
"parity+highlow": [{1, 3}, {5}, {2}, {4, 6}],
"everything": [{1}, {2}, {3}, {4}, {5}, {6}],
}
print(f"{'information':>16} {'E[Z]':>8} {'E[(Y-Z)^2]':>12}")
for name, blocks in information.items():
Z = cond_exp(blocks)
print(f"{name:>16} {Z.mean():>8.4f} {((Y - Z)**2).mean():>12.4f}")
print("\nE[Z] = 3.5 always (tower). The residual error shrinks as information grows,")
print("reaching 0 only with full information.")
Conditional expectation as the best predictor — the projection property:
import numpy as np
omega = np.array([1, 2, 3, 4, 5, 6])
Y = omega.astype(float)
is_odd = omega % 2 == 1
# Any G-measurable Z is determined by two numbers: its value on odds and evens.
best, best_err = None, np.inf
for a in np.linspace(1, 6, 501):
for b in np.linspace(1, 6, 501):
Z = np.where(is_odd, a, b)
err = ((Y - Z) ** 2).mean()
if err < best_err:
best, best_err = (a, b), err
print(f"best constants by brute-force search: odd -> {best[0]:.3f}, even -> {best[1]:.3f}")
print(f"conditional expectation says : odd -> 3.000, even -> 4.000")
print(f"minimum mean squared error : {best_err:.4f}")
print("\nE[Y|G] is exactly the least-squares best predictor using only G.")
Radon–Nikodym derivatives, computed as a likelihood ratio:
import numpy as np
from scipy.stats import norm
# Two distributions on the real line, both absolutely continuous w.r.t. Lebesgue
P0 = norm(0, 1)
P1 = norm(1, 1)
xs = np.array([-2.0, -1.0, 0.0, 1.0, 2.0, 3.0])
print("dP1/dP0 (x) = f1(x) / f0(x):\n")
print(f"{'x':>6} {'f0':>10} {'f1':>10} {'dP1/dP0':>10}")
for x in xs:
print(f"{x:>6.1f} {P0.pdf(x):>10.5f} {P1.pdf(x):>10.5f} "
f"{P1.pdf(x)/P0.pdf(x):>10.5f}")
print("\nThis ratio IS the Neyman-Pearson likelihood ratio from §5 --")
print("a Radon-Nikodym derivative of one hypothesis with respect to the other.")
Absolute continuity failing — a discrete distribution has no Lebesgue density:
import numpy as np
# A distribution concentrated on {0, 1} has all its mass on a Lebesgue-null set
print("P puts mass 1 on the two-point set {0, 1}.")
print("Lebesgue measure of {0, 1} = 0.")
print("\nSo P is NOT absolutely continuous w.r.t. Lebesgue:")
print(" lambda(A) = 0 but P(A) = 1 for A = {0, 1}.")
print("=> no Lebesgue density exists.\n")
print("But w.r.t. COUNTING measure on {0,1}, the density is just the PMF:")
for x, p in [(0, 0.3), (1, 0.7)]:
print(f" dP/dcount ({x}) = {p}")
print("\nThe discrete/continuous split is a choice of reference measure,")
print("not a fundamental difference.")
Your turn
1. Is a discrete distribution absolutely continuous with respect to Lebesgue measure?
2. E[Y \mid \mathcal G] where \mathcal G = \{\varnothing, \Omega\}?
3. Why does conditional expectation need measure theory at all?
Solutions
1. No.
A discrete distribution puts all its mass on a countable set S. But every countable set has Lebesgue measure zero, so
\lambda(S) = 0 \qquad \text{while} \qquad P(S) = 1
That directly violates the requirement \lambda(A) = 0 \implies P(A) = 0. Hence P \not\ll \lambda, and by Radon–Nikodym no density with respect to Lebesgue measure can exist.
But the PMF is a density — with respect to counting measure on S:
P(A) = \sum_{x \in A}p(x) = \int_A p\,d(\text{count})
So "discrete" and "continuous" aren't different kinds of object. They're the same kind of object measured against different rulers, which is why the Lebesgue formulation of expectation (last-but-one lesson) needed no cases.
2. It's the constant E[Y].
The trivial \sigma-algebra carries no information, so the only \mathcal G-measurable random variables are constants (§9, lesson 1). Condition 2 with A = \Omega then forces
\int_\Omega Z\,dP = Z = \int_\Omega Y\,dP = E[Y]
Interpretation: with nothing observed, the best prediction of Y is its unconditional mean. This is the degenerate end of the projection picture — you're projecting onto the space of constants, and the least-squares best constant is the mean.
3. Because the elementary definition P(A \mid B) = P(A \cap B)/P(B) divides by zero for continuous conditioning variables.
If X is continuous, P(X = x) = 0 for every x, so E[Y \mid X = x] is undefined as a ratio — yet it's a quantity we use constantly (it is the regression function of §6).
The measure-theoretic definition sidesteps the division entirely. Rather than conditioning on a single null event, it conditions on a \sigma-algebra and requires only that averages match over every set in it:
\int_A Z\,dP = \int_A Y\,dP \quad \text{for all } A \in \mathcal G
No division, no null-set problem. Radon–Nikodym then guarantees such a Z exists.
The cost is that E[Y \mid X = x] is only defined up to null sets — its value at any single x is not determined, only its behaviour on sets of positive probability. That's the source of the Borel–Kolmogorov paradox, where conditioning on the same event described two different ways gives two different answers. The resolution is that "the event" was never the object being conditioned on; the \sigma-algebra was.
Check yourself in code
Compute E[Y \mid \mathcal G] for a die roll given parity, verify the tower property, and confirm it is the least-squares best predictor.
Print exactly this:
E[Y|odd] 3.0
E[Y|even] 4.0
tower holds: True
residual MSE 2.6667
Round every value to 4 decimal places. The tower property holds if E[E[Y \mid \mathcal G]] = E[Y].
import numpy as np
omega = np.array([1, 2, 3, 4, 5, 6])
Y = omega.astype(float)
is_odd = omega % 2 == 1
print("E[Y|odd]", round(Y[is_odd].mean(), 4))
# Print E[Y|even], build Z from the two block averages, check the tower
# property E[Z] == E[Y], and report the residual mean squared error.
import numpy as np
omega = np.array([1, 2, 3, 4, 5, 6])
Y = omega.astype(float)
is_odd = omega % 2 == 1
print("E[Y|odd]", round(Y[is_odd].mean(), 4))
print("E[Y|even]", round(Y[~is_odd].mean(), 4))
Z = np.where(is_odd, Y[is_odd].mean(), Y[~is_odd].mean())
print("tower holds:", bool(np.isclose(Z.mean(), Y.mean())))
print("residual MSE", round(((Y - Z) ** 2).mean(), 4))
The Radon–Nikodym theorem says that when one measure never puts mass where another sees none, it can be written as an integral of a density — so a PDF is a derivative with respect to Lebesgue measure, a PMF a derivative with respect to counting measure, and a likelihood ratio a derivative of one hypothesis with respect to another. Conditional expectation is that same theorem applied to information: the best \mathcal G-measurable prediction of Y, matching its average on everything \mathcal G can see.
That closes §9, and with it the foundations. Next: measuring information itself.