12. Characteristic functions

The moment generating function is powerful, but it has a quiet weakness: for some distributions it simply doesn't exist. Its more dependable cousin fixes that with one small change.

This lesson uses complex numbers — specifically i = \sqrt{-1} and Euler's formula e^{i\theta} = \cos\theta + i\sin\theta. It is also the most skippable lesson in the module: if complex numbers are unfamiliar, jump to §1.5 and come back before §3, which is the first place characteristic functions earn their keep. §3.2 re-derives the CLT sketch from scratch, so nothing downstream silently assumes you were here.

The problem with the MGF

Recall M_X(t) = E[e^{tX}]. That expectation is a sum or integral, and there's no guarantee it converges.

For a distribution with genuinely heavy tails, e^{tX} grows faster than the density decays, and the integral diverges for every t \ne 0. The MGF exists only at t = 0, where it's trivially 1 — useless.

The standard example is the Cauchy distribution:

f(x) = \frac{1}{\pi(1 + x^2)}

Its tails decay like 1/x^2, which is slow enough that even E[X] = \int x f(x)\,dx fails to converge. No mean, no variance, no MGF.

This isn't an exotic edge case. Cauchy arises naturally as the ratio of two independent standard Normals, and heavy-tailed distributions show up constantly in finance and physics. A tool that breaks on them is a tool you can't rely on for general theorems.

The fix: an imaginary exponent

Swap the real exponent for an imaginary one:

\boxed{\;\varphi_X(t) = E\big[e^{itX}\big]\;}

where i = \sqrt{-1}. This is the characteristic function. Same idea as the MGF, with an i tucked into the exponent.

Why that tiny change fixes everything

By Euler's formula:

e^{i\theta} = \cos\theta + i\sin\theta

So e^{i\theta} traces the unit circle in the complex plane. Its magnitude is

\big|e^{i\theta}\big| = \sqrt{\cos^2\theta + \sin^2\theta} = 1

for every real \theta — it never grows.

So e^{itX} always has magnitude exactly 1, no matter how large X gets. It's bounded, completely tame, however wild the tails. And the expectation of a bounded quantity always exists:

\big|\varphi_X(t)\big| = \big|E[e^{itX}]\big| \le E\big[\,\big|e^{itX}\big|\,\big] = E[1] = 1

The characteristic function exists for every real t, for absolutely any distribution. No conditions, no exceptions. Compare that with the MGF's "if the integral converges" and you see why theorists reach for it.

Equivalently, \varphi_X(t) = E[\cos(tX)] + i\,E[\sin(tX)] — both expectations are of bounded functions, so both are finite.

The properties carry over

Everything useful about the MGF survives, with the same proofs:

It always equals 1 at zero: \varphi_X(0) = E[e^0] = 1.

Uniqueness. If \varphi_X = \varphi_Y, then X and Y have the same distribution — and now this holds unconditionally, since both always exist.

Sums become products. For independent X and Y:

\varphi_{X+Y}(t) = \varphi_X(t)\,\varphi_Y(t)

Affine transforms: \varphi_{aX+b}(t) = e^{ibt}\varphi_X(at).

Moments, when they exist:

\varphi_X^{(k)}(0) = i^k E[X^k]

The i^k is the price of the imaginary exponent — for the mean you get \varphi'(0) = iE[X], so divide by i. And the qualifier matters: if the k-th moment doesn't exist (Cauchy again), \varphi still exists but isn't k-times differentiable at 0.

Why this is the tool for the CLT

Here is the payoff. Suppose X_1, \dots, X_n are independent and identically distributed, and you want to know the distribution of their standardised sum.

Directly, adding random variables means convolution — an integral for two variables, an n-fold nightmare for n. Via characteristic functions, it's multiplication:

\varphi_{X_1 + \cdots + X_n}(t) = \big[\varphi_X(t)\big]^n

Now the entire Central Limit Theorem becomes an exercise in Taylor expansion. Standardise so the mean is 0 and variance 1, and expand near t = 0:

\varphi_X(t) = 1 - \frac{t^2}{2} + o(t^2)

(The linear term vanishes because E[X] = 0; the quadratic coefficient is -\frac{1}{2}E[X^2] = -\frac{1}{2}.)

For the standardised sum S_n = \frac{1}{\sqrt n}\sum X_j:

\varphi_{S_n}(t) = \left[\varphi_X\!\left(\frac{t}{\sqrt n}\right)\right]^n = \left[1 - \frac{t^2}{2n} + o\!\left(\frac{1}{n}\right)\right]^n \longrightarrow e^{-t^2/2}

using \left(1 + \frac{c}{n}\right)^n \to e^c. And e^{-t^2/2} is precisely the characteristic function of the standard Normal.

By uniqueness (plus a continuity theorem), S_n converges in distribution to N(0,1). That's the Central Limit Theorem — and notice the proof never needed the MGF to exist, only a finite variance. We'll unpack the statement properly in §3; the machinery is here.

Worked example

Find the characteristic function of a Bernoulli(p), and use it to get the mean.

\varphi_X(t) = E[e^{itX}] = e^{i \cdot 0}(1-p) + e^{i \cdot 1 \cdot t}p = (1 - p) + pe^{it}

Exactly the MGF with t replaced by it — which is the general relationship: \varphi_X(t) = M_X(it) whenever M_X exists.

Check \varphi_X(0) = 1 - p + p = 1. ✓

Now the mean:

\varphi_X'(t) = ipe^{it} \implies \varphi_X'(0) = ip

Divide by i:

E[X] = \frac{\varphi_X'(0)}{i} = p \quad\checkmark

And now the case the MGF can't touch. The standard Cauchy has

\varphi_X(t) = e^{-|t|}

which is perfectly well defined everywhere — but notice it is not differentiable at t = 0 (the absolute value has a corner there). That non-differentiability is precisely the statement that the mean does not exist. The characteristic function doesn't hide the problem; it reports it honestly while still existing.

A striking consequence: for n iid Cauchy variables,

\varphi_{\bar X_n}(t) = \left[e^{-|t|/n}\right]^n = e^{-|t|}

The sample mean of n Cauchys has exactly the same distribution as a single one. Averaging does nothing — no convergence, no shrinking spread. This is what it looks like when the Law of Large Numbers' finite-mean assumption is violated, and it's a one-line proof with characteristic functions.

Doing it in Python

Python handles complex numbers natively, so the definition transcribes directly:

import numpy as np

def char_fn(values, probs, t):
    """phi(t) = E[e^{itX}] for a discrete distribution."""
    return sum(p * np.exp(1j * t * x) for x, p in zip(values, probs))

# Bernoulli(0.3)
values, probs = [0, 1], [0.7, 0.3]

for t in [0.0, 0.5, 1.0, 2.0]:
    phi = char_fn(values, probs, t)
    print(f"phi({t}) = {phi:.4f}   |phi| = {abs(phi):.4f}")

Every magnitude comes out \le 1, as the bound guarantees.

The Cauchy comparison makes the whole lesson concrete — the MGF integral blows up while the characteristic function stays finite:

import numpy as np
from scipy.integrate import quad

f = lambda x: 1 / (np.pi * (1 + x**2))        # standard Cauchy density

# The MGF integrand e^{tx} f(x) diverges for any t != 0.
t = 1.0
mgf_partial = [quad(lambda x: np.exp(t * x) * f(x), -L, L)[0] for L in (10, 20, 40, 80)]
print("MGF integral over [-L, L], t = 1:")
for L, v in zip((10, 20, 40, 80), mgf_partial):
    print(f"   L = {L:>3}: {v:,.1f}")
print("  -> growing without bound: the MGF does not exist\n")

# The characteristic function integrand is bounded; the integral converges to e^{-|t|}.
re, _ = quad(lambda x: np.cos(t * x) * f(x), -np.inf, np.inf)
print(f"phi({t}) numerically = {re:.6f}")
print(f"theory  e^-|t|      = {np.exp(-abs(t)):.6f}")

And the averaging result — the sample mean of Cauchys refuses to settle down, while Normals converge as expected:

import numpy as np

rng = np.random.default_rng(1)
for n in (10, 100, 1_000, 10_000):
    cauchy_mean = rng.standard_cauchy(n).mean()
    normal_mean = rng.standard_normal(n).mean()
    print(f"n = {n:>6}:  Cauchy mean = {cauchy_mean:>10.3f}   Normal mean = {normal_mean:>7.3f}")

Your turn

1. What is \varphi_X(t) for a constant X = c?

2. Given |\varphi_X(t)| \le 1 always, what does |\varphi_X(t)| = 1 for all t tell you?

3. X, Y are independent standard Normals, each with \varphi(t) = e^{-t^2/2}. Find the distribution of X + Y.

Solutions

1. With all mass at c:

\varphi_X(t) = E[e^{itX}] = e^{itc}

Magnitude 1 for every t, since it sits on the unit circle. And \varphi'(0) = ic, so E[X] = c. ✓

2. It means X is degenerate — constant with probability 1.

The reasoning: |\varphi(t)| = |E[e^{itX}]| = 1 requires the averaged unit vectors e^{itX} to point in the same direction for all values X takes. Any spread in X makes them point differently, and averaging vectors on a circle that don't align gives something strictly inside it. So no spread is allowed. (Part 1 is the converse: a constant does achieve magnitude 1.)

3. By independence, multiply:

\varphi_{X+Y}(t) = e^{-t^2/2} \cdot e^{-t^2/2} = e^{-t^2}

Now recognise it. A Normal with mean 0 and variance \sigma^2 has \varphi(t) = e^{-\sigma^2 t^2/2}. Matching e^{-t^2} = e^{-2t^2/2} gives \sigma^2 = 2.

So X + Y \sim N(0, 2), and by uniqueness that's the only possible answer.

This is the general fact that sums of independent Normals are Normal, with variances adding — proved here in two lines instead of by convolving two Gaussian integrals.

Check yourself in code

Compute the characteristic function of a Bernoulli(0.3) at several values of t, confirm the magnitude bound holds everywhere, and recover E[X] from \varphi'(0)/i.

Print exactly this:

phi(0) = 1.0
max |phi| over t: 1.0
E[X] from phi'(0)/i = 0.3

Round every value to 4 decimal places.

import numpy as np

values, probs = [0, 1], [0.7, 0.3]
phi = lambda t: sum(p * np.exp(1j * t * x) for x, p in zip(values, probs))

print("phi(0) =", round(phi(0.0).real, 4))

# Check |phi(t)| <= 1 across a grid of t, then recover E[X] as phi'(0) / i
# using a central difference: (phi(h) - phi(-h)) / (2h).

import numpy as np

values, probs = [0, 1], [0.7, 0.3]
phi = lambda t: sum(p * np.exp(1j * t * x) for x, p in zip(values, probs))

print("phi(0) =", round(phi(0.0).real, 4))

grid = np.linspace(-50, 50, 20_001)
print("max |phi| over t:", round(max(abs(phi(t)) for t in grid), 4))

h = 1e-5
derivative = (phi(h) - phi(-h)) / (2 * h)
print("E[X] from phi'(0)/i =", round((derivative / 1j).real, 4))

---

Same idea as the moment generating function, but the imaginary exponent keeps it bounded — so it exists for every distribution, without conditions. That unconditional existence, plus turning sums into products, is what makes it the standard tool for proving limit theorems.

Next: the handful of named discrete distributions that describe most of the counting problems you'll ever meet.