Cross entropy · Probability & Statistics

Entropy assumes you know the true distribution p and can build the optimal code for it. But in practice you have a model q — an estimate, a prediction, a fitted network.

Cross entropy is the cost of using q's code on data that actually comes from p.

The definition

$H(p, q) = -\sum_x p(x)\log q(x) = E_p\left[\log\frac{1}{q(X)}\right]$

Note carefully which distribution does what:

p supplies the weights — the frequencies with which outcomes actually occur.
q supplies the code lengths — \log\frac{1}{q(x)} bits assigned to outcome x.

So cross entropy is the average code length you actually pay when you designed the code for q but nature draws from p.

Setting q = p recovers the entropy: H(p, p) = H(p).

It is never smaller than the entropy

$H(p, q) \ge H(p), \qquad \text{with equality iff } q = p$

Gibbs' inequality. Using the wrong distribution always costs you, and the penalty is exactly the KL divergence of the next lesson:

$H(p, q) = H(p) + D_{KL}(p \parallel q)$

That decomposition is worth memorising. Cross entropy splits into:

H(p) — the irreducible cost, fixed by nature.
D_{KL}(p \parallel q) \ge 0 — the avoidable cost of your model being wrong.

Minimising cross entropy over q is therefore the same as minimising the KL divergence, since H(p) doesn't depend on q at all. That is why cross entropy is a loss function.

Asymmetry

$H(p, q) \ne H(q, p) \quad \text{in general}$

The two arguments play completely different roles, so there's no reason to expect symmetry — and it matters. In machine learning, p is the data and q the model, always in that order.

Why it can be infinite

If q(x) = 0 for some x with p(x) > 0, then

$H(p, q) = \infty$

Your model declared an outcome impossible, and it happened. The code has no symbol for it, so the cost is unbounded.

This is why models never output exact zeros. A classifier predicting probability 0 for the true class incurs infinite loss, and a single such example destroys the average. Practical fixes:

Clipping: bound predictions to [\varepsilon, 1-\varepsilon].
Smoothing: add a small constant to counts (Laplace smoothing, §7's Beta(1,1) prior in disguise).
Softmax: by construction outputs are strictly positive, which is a large part of why it's used.

As a loss function

Cross entropy is the standard loss for classification. For a single example with true label y and predicted probabilities \hat y:

$L = -\sum_c y_c\log \hat y_c$

With one-hot labels only the true class survives:

$L = -\log \hat y_{\text{true}}$

"The negative log probability you assigned to the right answer." Confident and correct → near 0. Confident and wrong → very large.

For binary classification this specialises to binary cross entropy (log loss):

$L = -\big[y\log\hat y + (1-y)\log(1-\hat y)\big]$

which you have already seen: it is exactly the negative log-likelihood of logistic regression from §6. Minimising cross entropy is maximum likelihood estimation (§4) — the same procedure under two names.

Why cross entropy rather than squared error for classification? Two reasons. It's the correct likelihood for categorical data, and its gradient with a softmax output is simply \hat y - y — clean, and free of the vanishing gradients that squared error suffers when predictions saturate.

Proper scoring

Cross entropy is a proper scoring rule: it is minimised in expectation by reporting your honest probabilities.

A model cannot improve its expected loss by overstating confidence. Predicting 0.99 when you believe 0.8 raises your expected cross entropy. That property is what makes it suitable for training calibrated probabilistic models rather than merely accurate classifiers.

Accuracy, by contrast, is not proper — it only cares which side of 0.5 you land on, so it gives no incentive to be honest about how sure you are.

Worked example

True distribution p = (0.7, 0.2, 0.1). Compare two models.

Model A: q_A = (0.6, 0.3, 0.1) — reasonably close. Model B: q_B = (0.1, 0.2, 0.7) — badly wrong.

First the entropy — the floor:

$H(p) = -(0.7\log_2 0.7 + 0.2\log_2 0.2 + 0.1\log_2 0.1) \approx 1.157 \text{ bits}$

Model A:

$H(p, q_A) = -(0.7\log_2 0.6 + 0.2\log_2 0.3 + 0.1\log_2 0.1) \approx 1.196 \text{ bits}$

Model B:

$H(p, q_B) = -(0.7\log_2 0.1 + 0.2\log_2 0.2 + 0.1\log_2 0.7) \approx 2.841 \text{ bits}$

Both exceed H(p) = 1.157, as Gibbs' inequality requires.

Model A costs 0.039 extra bits per symbol — a good model. Model B costs 1.684 extra bits, more than doubling the message length.

Notice where B's cost comes from: it assigns probability 0.1 to the outcome that occurs 70% of the time, so most observations are charged \log_2(1/0.1) = 3.32 bits each. Being confidently wrong about the common case is what cross entropy punishes hardest.

Doing it in Python

Cross entropy and the decomposition:

import numpy as np

def H(p, base=2):
    p = np.asarray(p, float)
    p = p[p > 0]
    return -np.sum(p * np.log(p) / np.log(base))

def cross_entropy(p, q, base=2):
    p, q = np.asarray(p, float), np.asarray(q, float)
    mask = p > 0
    return -np.sum(p[mask] * np.log(q[mask]) / np.log(base))

def kl(p, q, base=2):
    return cross_entropy(p, q, base) - H(p, base)

p = np.array([0.7, 0.2, 0.1])
models = {
    "perfect  (= p)":  [0.7, 0.2, 0.1],
    "close":           [0.6, 0.3, 0.1],
    "uniform":         [1/3, 1/3, 1/3],
    "badly wrong":     [0.1, 0.2, 0.7],
}
print(f"H(p) = {H(p):.4f} bits  (the floor)\n")
print(f"{'model':>18} {'H(p,q)':>10} {'KL':>10} {'check':>10}")
for name, q in models.items():
    ce, d = cross_entropy(p, q), kl(p, q)
    print(f"{name:>18} {ce:>10.4f} {d:>10.4f} {H(p) + d:>10.4f}")

print("\nH(p,q) = H(p) + KL(p||q) exactly, and KL >= 0 always.")

Asymmetry, and the infinite case:

import numpy as np

def cross_entropy(p, q, base=2):
    p, q = np.asarray(p, float), np.asarray(q, float)
    mask = p > 0
    if np.any(q[mask] == 0):
        return float("inf")
    return -np.sum(p[mask] * np.log(q[mask]) / np.log(base))

p = [0.7, 0.2, 0.1]
q = [0.1, 0.2, 0.7]
print(f"H(p, q) = {cross_entropy(p, q):.4f}")
print(f"H(q, p) = {cross_entropy(q, p):.4f}   <- not the same")

# A model that rules out something that happens
q_zero = [0.5, 0.5, 0.0]
print(f"\nmodel assigns 0 to an outcome with p = 0.1:")
print(f"  H(p, q) = {cross_entropy(p, q_zero)}")
print("  -> a single 'impossible' event that occurs costs infinity.")

Cross entropy as a classification loss, and why confidence is punished:

import numpy as np

def binary_ce(y, yhat, eps=1e-15):
    yhat = np.clip(yhat, eps, 1 - eps)
    return -(y * np.log(yhat) + (1 - y) * np.log(1 - yhat))

print("True label is 1. Loss as the prediction moves:\n")
print(f"{'prediction':>12} {'loss':>10}")
for yhat in (0.99, 0.9, 0.7, 0.5, 0.3, 0.1, 0.01, 0.0001):
    print(f"{yhat:>12} {binary_ce(1, yhat):>10.4f}")

print("\nConfident and right -> near 0. Confident and WRONG -> enormous.")
print("That asymmetry is what forces models to be honest about uncertainty.")

Proper scoring — honesty minimises expected loss:

import numpy as np

def binary_ce(y, yhat, eps=1e-15):
    yhat = np.clip(yhat, eps, 1 - eps)
    return -(y * np.log(yhat) + (1 - y) * np.log(1 - yhat))

true_p = 0.7          # the event really happens 70% of the time

print(f"The event occurs with probability {true_p}.")
print("Expected cross-entropy loss for various REPORTED probabilities:\n")
print(f"{'reported':>10} {'expected loss':>15}")
best, best_loss = None, np.inf
for reported in np.arange(0.05, 1.0, 0.05):
    loss = true_p * binary_ce(1, reported) + (1 - true_p) * binary_ce(0, reported)
    if loss < best_loss:
        best, best_loss = reported, loss
    if round(reported, 2) in (0.1, 0.3, 0.5, 0.7, 0.9):
        print(f"{reported:>10.2f} {loss:>15.4f}")

print(f"\nminimised at reported = {best:.2f} -- the TRUE probability.")
print("You cannot lower your expected loss by exaggerating confidence.")

And the equivalence with maximum likelihood:

import numpy as np
from scipy.optimize import minimize

rng = np.random.default_rng(0)
n = 500
x = rng.normal(0, 1, n)
true_b = np.array([0.5, 1.5])
X = np.column_stack([np.ones(n), x])
y = (rng.random(n) < 1 / (1 + np.exp(-(X @ true_b)))).astype(float)

def mean_cross_entropy(b):
    eta = X @ b
    yhat = 1 / (1 + np.exp(-eta))
    yhat = np.clip(yhat, 1e-12, 1 - 1e-12)
    return -np.mean(y * np.log(yhat) + (1 - y) * np.log(1 - yhat))

def neg_loglik(b):
    eta = X @ b
    return -np.sum(y * eta - np.logaddexp(0, eta))

ce_fit = minimize(mean_cross_entropy, [0.0, 0.0]).x
ml_fit = minimize(neg_loglik, [0.0, 0.0]).x

print("minimising cross entropy :", ce_fit.round(4))
print("maximising likelihood    :", ml_fit.round(4))
print("true coefficients        :", true_b)
print("\nIdentical -- they are the same optimisation, differing only by a")
print("constant factor of n.")

Your turn

1. p = (0.5, 0.5), q = (0.9, 0.1). Find H(p, q) in bits.

2. Why does cross entropy become infinite when q assigns zero to a possible outcome?

3. A classifier predicts 0.99 for the true class. What's the loss? What if it predicts 0.01?

Solutions

$H(p, q) = -\left(0.5\log_2 0.9 + 0.5\log_2 0.1\right)$

$= -0.5(-0.152) - 0.5(-3.322) = 0.076 + 1.661 = 1.737 \text{ bits}$

Compare with H(p) = 1 bit. The excess, D_{KL}(p \parallel q) = 0.737 bits, is the price of modelling a fair coin as a 90/10 coin.

Half the time the model is charged 3.32 bits for an outcome it thought was rare — and that half dominates the average.

2. Because the code length for outcome x is \log\frac{1}{q(x)}, and

$\lim_{q \to 0^+}\log\frac{1}{q} = \infty$

If q(x) = 0 but p(x) > 0, the model has assigned no code at all to something that genuinely occurs. There is no finite number of bits that represents it, so the expected cost is infinite.

Read as a probability statement: the model declared the event impossible, and it happened. No amount of evidence can rescue a model that assigned zero — the same irreversibility as §7's Cromwell's rule, since \log 0 = -\infty is the log-scale version of multiplying by zero.

In practice this is why predictions are clipped or smoothed: a single confidently wrong prediction would otherwise make the average loss infinite regardless of how well the model does on everything else.

3. For a one-hot label the loss is -\log \hat y_{\text{true}}.

Predicting 0.99:

$L = -\log_2(0.99) \approx 0.0145 \text{ bits} \qquad (\text{or } -\ln 0.99 \approx 0.01 \text{ nats})$

Nearly zero — confident and correct.

Predicting 0.01:

$L = -\log_2(0.01) \approx 6.64 \text{ bits} \qquad (\approx 4.61 \text{ nats})$

About 460 times larger. The loss is unbounded below zero probability and grows without limit as confidence in the wrong answer increases.

That steep asymmetry is deliberate: it makes overconfidence far more expensive than mere inaccuracy, which is precisely what you want from a model that should report honest probabilities.

Check yourself in code

Compute cross entropy for two models against the same truth and confirm the decomposition H(p,q) = H(p) + D_{KL}(p \parallel q).

Print exactly this:

H(p) 1.1568
H(p,q) close 1.1955
H(p,q) wrong 2.8412
decomposition holds: True

Use p = (0.7, 0.2, 0.1), a close model (0.6, 0.3, 0.1) and a wrong model (0.1, 0.2, 0.7), base-2 logarithms, and round to 4 decimal places.

import numpy as np

def H(p):
    p = np.asarray(p, float)
    p = p[p > 0]
    return -np.sum(p * np.log2(p))

def cross_entropy(p, q):
    p, q = np.asarray(p, float), np.asarray(q, float)
    mask = p > 0
    return -np.sum(p[mask] * np.log2(q[mask]))

p = [0.7, 0.2, 0.1]
print("H(p)", round(H(p), 4))

# Print the two cross entropies, then verify H(p,q) - H(p) equals the KL
# divergence sum p*log2(p/q) for both models.

import numpy as np

def H(p):
    p = np.asarray(p, float)
    p = p[p > 0]
    return -np.sum(p * np.log2(p))

def cross_entropy(p, q):
    p, q = np.asarray(p, float), np.asarray(q, float)
    mask = p > 0
    return -np.sum(p[mask] * np.log2(q[mask]))

def kl(p, q):
    p, q = np.asarray(p, float), np.asarray(q, float)
    mask = p > 0
    return np.sum(p[mask] * np.log2(p[mask] / q[mask]))

p = [0.7, 0.2, 0.1]
close, wrong = [0.6, 0.3, 0.1], [0.1, 0.2, 0.7]

print("H(p)", round(H(p), 4))
print("H(p,q) close", round(cross_entropy(p, close), 4))
print("H(p,q) wrong", round(cross_entropy(p, wrong), 4))
print("decomposition holds:",
      all(np.isclose(cross_entropy(p, q), H(p) + kl(p, q)) for q in (close, wrong)))

Cross entropy is the cost of coding data from p with a code built for q. It can never fall below H(p), and the gap is exactly the KL divergence — so minimising it over models is minimising that divergence. It's the standard classification loss, identical to negative log-likelihood, and it goes to infinity for a model that rules out something that happens.

Next: that gap, studied on its own.

61. Cross entropy