KL divergence · Probability & Statistics

The gap between cross entropy and entropy has a name and a life of its own.

$D_{KL}(p \parallel q) = H(p, q) - H(p) = \sum_x p(x)\log\frac{p(x)}{q(x)}$

Equivalently, and often more usefully:

$D_{KL}(p \parallel q) = E_p\left[\log\frac{p(X)}{q(X)}\right]$

It measures how much information is lost when q is used to approximate p — the wasted bits from coding with the wrong model.

Key properties

Non-negative: D_{KL}(p \parallel q) \ge 0, with equality iff p = q.

This is Gibbs' inequality, and it follows from Jensen (§9): \log is concave, so

$-D_{KL}(p \parallel q) = E_p\left[\log\frac{q}{p}\right] \le \log E_p\left[\frac{q}{p}\right] = \log\sum_x q(x) = \log 1 = 0$

Asymmetric: D_{KL}(p \parallel q) \ne D_{KL}(q \parallel p) in general.

Not a metric. It fails symmetry and the triangle inequality, so it is called a divergence, not a distance. (If you need a genuine metric, the Jensen–Shannon divergence — a symmetrised, smoothed version — has a square root that is one.)

Infinite when q(x) = 0 but p(x) > 0, for the same reason cross entropy was.

Additive for independence: D_{KL}(p_1p_2 \parallel q_1q_2) = D_{KL}(p_1 \parallel q_1) + D_{KL}(p_2 \parallel q_2).

The asymmetry is not a defect

The two directions ask genuinely different questions, and choosing between them is a modelling decision.

Forward KL, D_{KL}(p \parallel q) — weighted by the true p. The penalty is large wherever p has mass and q does not. So minimising it makes q cover everything p does:

Mass-covering / mean-seeking. Fitting a unimodal q to a bimodal p spreads q across both modes, including the empty valley between them.

This is what maximum likelihood does — minimising forward KL to the empirical distribution is maximum likelihood (§4).

Reverse KL, D_{KL}(q \parallel p) — weighted by the model q. The penalty is large wherever q has mass and p does not, but q pays nothing for ignoring regions where it puts no mass. So minimising it makes q avoid regions p rules out:

Mode-seeking / zero-forcing. The same unimodal q locks onto one mode and ignores the other entirely.

This is what variational inference does, and it explains a well-known failure mode: variational posteriors are often too narrow, understating uncertainty because reverse KL rewards staying safely inside the true distribution's support.

Neither is "correct". Forward KL says "don't be surprised by the data"; reverse KL says "don't predict nonsense". Which you want depends on the cost of each error.

Where it appears

Maximum likelihood = minimising forward KL to the empirical distribution.
Variational inference minimises reverse KL between an approximate and true posterior. The ELBO is exactly that objective rearranged.
Mutual information is a KL divergence — next lesson.
Model selection: AIC is derived as an estimate of expected KL divergence between the fitted model and the truth.
Hypothesis testing: the expected log-likelihood ratio under H_0 is a KL divergence, which is why it governs the asymptotic power of tests (§5).

A closed form worth knowing

For two Normals:

$D_{KL}\big(N(\mu_1,\sigma_1^2) \parallel N(\mu_2,\sigma_2^2)\big) = \log\frac{\sigma_2}{\sigma_1} + \frac{\sigma_1^2 + (\mu_1-\mu_2)^2}{2\sigma_2^2} - \frac12$

Two things fall out. With equal variances it reduces to \frac{(\mu_1-\mu_2)^2}{2\sigma^2} — proportional to the squared distance between means, so KL grows quadratically as the distributions separate. And it is visibly asymmetric in \sigma_1, \sigma_2.

Worked example

p = (0.5, 0.5), q = (0.9, 0.1). Compute both directions.

Forward:

$D_{KL}(p \parallel q) = 0.5\log_2\frac{0.5}{0.9} + 0.5\log_2\frac{0.5}{0.1}$ $= 0.5(-0.848) + 0.5(2.322) = -0.424 + 1.161 = 0.737 \text{ bits}$

Reverse:

$D_{KL}(q \parallel p) = 0.9\log_2\frac{0.9}{0.5} + 0.1\log_2\frac{0.1}{0.5}$ $= 0.9(0.848) + 0.1(-2.322) = 0.763 - 0.232 = 0.531 \text{ bits}$

Different: 0.737 vs 0.531. Both are positive, as required, and neither is "the" divergence between these distributions.

The forward direction is larger because p puts substantial weight (0.5) on the outcome q considers rare, and that term contributes 1.161 bits on its own. Reverse KL weights by q, which only gives that outcome weight 0.1, so the same mismatch costs less.

The extreme case. If q = (1.0, 0.0):

$D_{KL}(p \parallel q) = \infty \qquad \text{but} \qquad D_{KL}(q \parallel p) = \log_2\frac{1}{0.5} = 1 \text{ bit}$

One direction is infinite and the other is perfectly finite. That is the asymmetry at its starkest, and it's exactly the mode-seeking behaviour: reverse KL happily lets q collapse onto a single outcome, while forward KL forbids it absolutely.

Doing it in Python

Both directions, and the non-negativity:

import numpy as np

def kl(p, q, base=2):
    p, q = np.asarray(p, float), np.asarray(q, float)
    mask = p > 0
    if np.any(q[mask] == 0):
        return float("inf")
    return np.sum(p[mask] * np.log(p[mask] / q[mask]) / np.log(base))

pairs = [
    ("(.5,.5) vs (.9,.1)",  [0.5, 0.5], [0.9, 0.1]),
    ("identical",           [0.5, 0.5], [0.5, 0.5]),
    ("(.5,.5) vs (1,0)",    [0.5, 0.5], [1.0, 0.0]),
    ("(.7,.2,.1) vs unif",  [0.7, 0.2, 0.1], [1/3, 1/3, 1/3]),
]
print(f"{'pair':>22} {'KL(p||q)':>12} {'KL(q||p)':>12}")
for name, p, q in pairs:
    print(f"{name:>22} {kl(p, q):>12.4f} {kl(q, p):>12.4f}")

print("\nAlways >= 0, zero only when identical, and rarely symmetric.")

Mass-covering versus mode-seeking — the clearest way to see why the direction matters:

import numpy as np
from scipy.stats import norm
from scipy.optimize import minimize

grid = np.linspace(-8, 12, 4000)
dx = grid[1] - grid[0]

# A bimodal truth
p = 0.5 * norm.pdf(grid, 0, 1) + 0.5 * norm.pdf(grid, 6, 1)
p /= p.sum() * dx

def divergence(params, forward):
    mu, log_s = params
    q = norm.pdf(grid, mu, np.exp(log_s))
    q = np.clip(q, 1e-300, None)
    if forward:
        return np.sum(p * np.log(p / q)) * dx          # KL(p || q)
    return np.sum(q * np.log(q / np.clip(p, 1e-300, None))) * dx   # KL(q || p)

fwd = minimize(divergence, [3.0, 0.0], args=(True,)).x
rev = minimize(divergence, [3.0, 0.0], args=(False,)).x

print("true distribution: equal mixture of N(0,1) and N(6,1)\n")
print(f"forward KL(p||q) fit: mu = {fwd[0]:6.3f}, sigma = {np.exp(fwd[1]):5.3f}")
print(f"reverse KL(q||p) fit: mu = {rev[0]:6.3f}, sigma = {np.exp(rev[1]):5.3f}")
print("\nForward covers BOTH modes (wide, centred between them).")
print("Reverse locks onto ONE mode (narrow) -- and ignores the other.")

The Normal closed form, checked numerically:

import numpy as np
from scipy.stats import norm

def kl_normal(m1, s1, m2, s2):
    return np.log(s2 / s1) + (s1**2 + (m1 - m2) ** 2) / (2 * s2**2) - 0.5

grid = np.linspace(-30, 30, 400_000)
dx = grid[1] - grid[0]

def kl_numeric(m1, s1, m2, s2):
    p = norm.pdf(grid, m1, s1)
    q = np.clip(norm.pdf(grid, m2, s2), 1e-300, None)
    return np.sum(p * np.log(p / q)) * dx

cases = [(0, 1, 0, 1), (0, 1, 1, 1), (0, 1, 0, 2), (0, 2, 0, 1), (0, 1, 3, 1)]
print(f"{'N(m1,s1) -> N(m2,s2)':>24} {'closed form':>13} {'numeric':>10}")
for m1, s1, m2, s2 in cases:
    print(f"{f'N({m1},{s1}) -> N({m2},{s2})':>24} "
          f"{kl_normal(m1, s1, m2, s2):>13.6f} {kl_numeric(m1, s1, m2, s2):>10.6f}")

print(f"\nasymmetry: KL(N(0,1)||N(0,2)) = {kl_normal(0,1,0,2):.4f}, "
      f"reversed = {kl_normal(0,2,0,1):.4f}")

Maximum likelihood as forward-KL minimisation:

import numpy as np
from scipy.stats import norm
from scipy.optimize import minimize

rng = np.random.default_rng(0)
data = rng.normal(3.0, 2.0, 5_000)

# Maximum likelihood
def neg_loglik(params):
    mu, log_s = params
    return -np.sum(norm.logpdf(data, mu, np.exp(log_s)))

# Forward KL to the empirical distribution (same thing, up to a constant)
def forward_kl(params):
    mu, log_s = params
    return -np.mean(norm.logpdf(data, mu, np.exp(log_s)))

ml = minimize(neg_loglik, [0.0, 0.0]).x
kl_fit = minimize(forward_kl, [0.0, 0.0]).x

print(f"maximum likelihood : mu {ml[0]:.4f}, sigma {np.exp(ml[1]):.4f}")
print(f"minimum forward KL : mu {kl_fit[0]:.4f}, sigma {np.exp(kl_fit[1]):.4f}")
print(f"sample moments     : mu {data.mean():.4f}, sigma {data.std():.4f}")
print("\nIdentical -- MLE IS forward-KL minimisation against the empirical data.")

Your turn

1. D_{KL}(p \parallel p) = ?

2. Why is KL divergence not a distance metric?

3. You're fitting a unimodal model to bimodal data. What happens under forward KL versus reverse KL?

Solutions

1. Zero.

$D_{KL}(p \parallel p) = \sum_x p(x)\log\frac{p(x)}{p(x)} = \sum_x p(x)\log 1 = 0$

And by Gibbs' inequality this is the only case where it vanishes: $D_{KL} = 0 \iff p = q$. So while KL isn't a metric, it does correctly identify when two distributions are the same.

2. It fails two of the three metric axioms.

Symmetry fails: D_{KL}(p \parallel q) \ne D_{KL}(q \parallel p), as the worked example showed (0.737 vs 0.531). A distance must not depend on which point you start from.

The triangle inequality fails: there are distributions with D_{KL}(p \parallel r) > D_{KL}(p \parallel q) + D_{KL}(q \parallel r).

It does satisfy non-negativity and the identity of indiscernibles, which is why it's still a useful measure of dissimilarity — just not one you can do geometry with.

If you need a genuine metric: the Jensen–Shannon divergence

$JSD(p, q) = \tfrac12 D_{KL}(p \parallel m) + \tfrac12 D_{KL}(q \parallel m), \qquad m = \tfrac{p+q}{2}$

is symmetric, always finite (the mixture m can't be zero where either is positive), and \sqrt{JSD} satisfies the triangle inequality.

3. They give visibly different fits, and the difference is systematic.

Forward KL, D_{KL}(p \parallel q) — mass-covering. The sum is weighted by p, so any region where p has mass and q has little is heavily penalised (\log(p/q) blows up). To avoid that, q stretches to cover both modes, ending up wide and centred in the valley between them — placing most of its mass where the data almost never falls.

Reverse KL, D_{KL}(q \parallel p) — mode-seeking. Now the sum is weighted by q, so q is only penalised where it has mass. It pays nothing for ignoring a mode entirely. The optimum is to collapse onto one mode, fitting it tightly.

The code above shows exactly this: forward gives \mu \approx 3 with a large \sigma; reverse gives \mu near 0 or 6 with \sigma \approx 1.

Practical consequence: variational inference minimises reverse KL, which is why variational posteriors are notoriously overconfident — they under-report uncertainty by design. If you need calibrated uncertainty, that's a known limitation to work around (with richer approximating families, or with MCMC instead).

Check yourself in code

Compute KL divergence in both directions and confirm it is non-negative, asymmetric, and zero only for identical distributions.

Print exactly this:

KL(p||q) 0.737
KL(q||p) 0.531
asymmetric: True
KL(p||p) 0.0

Use p = (0.5, 0.5) and q = (0.9, 0.1) with base-2 logarithms, rounded to 4 decimal places.

import numpy as np

def kl(p, q):
    p, q = np.asarray(p, float), np.asarray(q, float)
    mask = p > 0
    return np.sum(p[mask] * np.log2(p[mask] / q[mask]))

p, q = [0.5, 0.5], [0.9, 0.1]
print("KL(p||q)", round(kl(p, q), 4))

# Print KL(q||p), whether the two differ, and KL(p||p).

import numpy as np

def kl(p, q):
    p, q = np.asarray(p, float), np.asarray(q, float)
    mask = p > 0
    return np.sum(p[mask] * np.log2(p[mask] / q[mask]))

p, q = [0.5, 0.5], [0.9, 0.1]
print("KL(p||q)", round(kl(p, q), 4))
print("KL(q||p)", round(kl(q, p), 4))
print("asymmetric:", round(kl(p, q), 4) != round(kl(q, p), 4))
print("KL(p||p)", round(kl(p, p), 4))

KL divergence is the expected excess bits from modelling p as q. It's non-negative, zero only when the distributions match, and asymmetric — and the asymmetry is a feature: forward KL covers the truth's mass (maximum likelihood), reverse KL seeks one mode and understates uncertainty (variational inference).

Next: KL divergence applied to the joint versus the product of marginals, which measures how much two variables tell you about each other.

62. KL divergence