Independence of random variables · Probability & Statistics

In §0 we defined independence for events. Now we lift it to random variables, which is a stronger requirement: it must hold for every pair of values at once.

The definition

X and Y are independent if their joint distribution factors into the product of their marginals:

$p(x, y) = p_X(x)\,p_Y(y) \quad \text{for all } x, y$

$f(x, y) = f_X(x)\,f_Y(y) \quad \text{for all } x, y$

Equivalently, wherever the conditioning is defined:

$f_{Y|X}(y \mid x) = f_Y(y)$

Learning X tells you nothing about Y — conditioning changes nothing. The conditional distribution is the same slice no matter where you slice.

The phrase for all x, y is doing real work. A single coincidental factorisation proves nothing; failure at one pair kills independence entirely.

What it buys you

Independence is not just a description — it's a licence to compute.

Probabilities multiply:

$P(X \in A, \; Y \in B) = P(X \in A)P(Y \in B)$

Expectations of products factor:

$E[XY] = E[X]\,E[Y]$

Variances add:

$\operatorname{Var}(X + Y) = \operatorname{Var}(X) + \operatorname{Var}(Y)$

MGFs multiply: M_{X+Y}(t) = M_X(t)M_Y(t) — the identity that powered §1.

That variance rule is the one to hold onto. Expectations always add, for any X and Y whatsoever. Variances add only when the variables are uncorrelated. The general formula carries a correction term:

$\operatorname{Var}(X + Y) = \operatorname{Var}(X) + \operatorname{Var}(Y) + 2\operatorname{Cov}(X, Y)$

and independence forces that covariance to zero. Covariance is the next lesson; this is where it comes from.

Independent vs. uncorrelated

E[XY] = E[X]E[Y] (equivalently, zero covariance) is called being uncorrelated. Independence implies it. The converse is false, and the counterexample is worth carrying around.

Let X be uniform on \{-1, 0, 1\} and set Y = X^2.

Y is a deterministic function of X — you could hardly be more dependent. Yet:

$E[X] = 0, \qquad E[XY] = E[X^3] = \frac{(-1)^3 + 0^3 + 1^3}{3} = 0$

$\operatorname{Cov}(X, Y) = E[XY] - E[X]E[Y] = 0 - 0 = 0$

Uncorrelated, but utterly dependent. Knowing X = 0 tells you Y = 0 with certainty.

The reason: covariance only detects linear association. The relationship here is a parabola — perfectly strong, perfectly symmetric, and invisible to a linear measure. This is why "uncorrelated" must never be read as "unrelated".

The one exception worth knowing: if (X, Y) are jointly Normal, then uncorrelated does imply independent. That's a special property of the multivariate Normal (lesson 5 of this module), not a general fact — and assuming it elsewhere is a classic error.

A quick structural test

Before computing anything, check the support — the set where the density is positive.

If X and Y are independent, the support must be a rectangle (a product region): if x_0 is possible for X and y_0 is possible for Y, then the pair (x_0, y_0) must be possible too, since f = f_Xf_Y > 0.

So any density defined on a triangle, a disc, or the region 0 < x < y < 1 is immediately not independent, whatever the algebra looks like. The constraint linking x to y is itself the dependence.

Worked example

f(x,y) = 4xy on the unit square. Independent?

From the previous lesson, f_X(x) = 2x and f_Y(y) = 2y on [0,1].

$f_X(x)f_Y(y) = (2x)(2y) = 4xy = f(x,y) \quad\checkmark$

Independent. The support is the unit square — a rectangle — and the density factors cleanly.

Now change the region: f(x,y) = 8xy on the triangle 0 < x < y < 1.

The support test settles it instantly: the region is a triangle, so x and y constrain each other and they cannot be independent.

Confirm it by computing. The marginal of X integrates y from x to 1 — note the limit depends on x, which is the dependence made visible:

$f_X(x) = \int_x^1 8xy\,dy = 8x\left[\frac{y^2}{2}\right]_x^1 = 4x(1 - x^2)$

$f_Y(y) = \int_0^y 8xy\,dx = 8y\left[\frac{x^2}{2}\right]_0^y = 4y^3$

$f_X(x)f_Y(y) = 16xy^3(1-x^2) \; \ne \; 8xy$

Not independent. ✓

The lesson: the formula 8xy looks perfectly factorable. It's the domain that creates the dependence, and only the support test catches that cheaply.

Doing it in Python

Testing independence is comparing the joint against the outer product of the marginals:

import numpy as np

def independent(joint, tol=1e-12):
    """Does the joint equal the outer product of its marginals?"""
    px = joint.sum(axis=1)
    py = joint.sum(axis=0)
    return np.allclose(joint, np.outer(px, py), atol=tol)

# Weather vs commute from the last lesson
weather_commute = np.array([[0.15, 0.10, 0.25],
                            [0.05, 0.40, 0.05]])
print("weather/commute independent?", independent(weather_commute))
print("outer product would be:\n",
      np.outer(weather_commute.sum(axis=1), weather_commute.sum(axis=0)).round(4))

# Two independent fair dice: the joint IS the outer product by construction
dice = np.outer(np.full(6, 1/6), np.full(6, 1/6))
print("\ntwo dice independent?", independent(dice))

The uncorrelated-but-dependent counterexample, made concrete:

import numpy as np

x = np.array([-1, 0, 1])
p = np.full(3, 1/3)
y = x**2                       # Y is a deterministic function of X

EX = (x * p).sum()
EY = (y * p).sum()
EXY = (x * y * p).sum()

print("E[X]   =", round(EX, 10))
print("E[Y]   =", round(EY, 4))
print("E[XY]  =", round(EXY, 10))
print("Cov    =", round(EXY - EX * EY, 10), "-> uncorrelated")
print("corr   =", round(np.corrcoef(np.repeat(x, 1000), np.repeat(y, 1000))[0, 1], 10))
print()
print("But Y is X**2 -- knowing X gives Y exactly. Dependent.")

And the variance rule, which is where independence actually pays:

import numpy as np

rng = np.random.default_rng(0)
n = 500_000

A = rng.standard_normal(n)
B = rng.standard_normal(n)          # independent of A
C = A                               # perfectly dependent on A

print("Var(A)          =", round(A.var(), 4))
print("Var(B)          =", round(B.var(), 4))
print()
print("independent:  Var(A + B) =", round((A + B).var(), 4), " (~2 = 1 + 1)")
print("dependent:    Var(A + C) =", round((A + C).var(), 4), " (~4, NOT 2)")
print()
print("E[A + B] =", round((A + B).mean(), 4), " E[A + C] =", round((A + C).mean(), 4))
print("-> expectations add either way; variances do not")

Your turn

1. X, Y are independent with E[X] = 2, \operatorname{Var}(X) = 3, E[Y] = 5, \operatorname{Var}(Y) = 4. Find E[X + Y], \operatorname{Var}(X+Y) and E[XY].

2. Is f(x,y) = x + y on the unit square a product of its marginals?

3. X is uniform on [0,1] and Y = 1 - X. Are they independent? Are they correlated?

Solutions

1. Expectation is linear regardless of dependence:

$E[X + Y] = 2 + 5 = 7$

Variances add because they're independent:

$\operatorname{Var}(X + Y) = 3 + 4 = 7$

And the product factors, also because of independence:

$E[XY] = E[X]E[Y] = 2 \times 5 = 10$

Without independence only the first would survive; the other two would need the covariance.

2. Compute the marginals:

$f_X(x) = \int_0^1 (x + y)\,dy = x + \tfrac{1}{2}$

and by symmetry f_Y(y) = y + \tfrac12. Then

$f_X(x)f_Y(y) = \left(x + \tfrac12\right)\left(y + \tfrac12\right) = xy + \tfrac{x}{2} + \tfrac{y}{2} + \tfrac14$

which is not x + y. So no — not independent.

The support is a perfectly good rectangle here, so the cheap test doesn't fire; this one genuinely needs the algebra. The additive form x+y is the giveaway — independence needs a product structure, and a sum can't be rearranged into one.

3. Not independent — as dependent as possible. Y is a deterministic function of X: knowing X = 0.3 gives Y = 0.7 with certainty. (The support is the diagonal line segment, not a rectangle, so the structural test catches it at a glance.)

And they are correlated — perfectly negatively. Using \operatorname{Var}(aX + b) = a^2\operatorname{Var}(X):

$\operatorname{Cov}(X, Y) = \operatorname{Cov}(X, 1 - X) = -\operatorname{Var}(X) = -\tfrac{1}{12}$

giving a correlation of exactly -1.

This contrasts sharply with the Y = X^2 example: both are deterministic relationships, but this one is linear, so covariance sees it perfectly, while the parabola was invisible.

Check yourself in code

Test two joint distributions for independence — the weather/commute table (dependent) and two fair dice (independent) — and demonstrate the uncorrelated-but-dependent case.

Print exactly this:

weather/commute independent: False
two dice independent: True
Cov(X, X^2) = 0.0
but Y is determined by X: True

import numpy as np

def independent(joint):
    px, py = joint.sum(axis=1), joint.sum(axis=0)
    return bool(np.allclose(joint, np.outer(px, py)))

weather_commute = np.array([[0.15, 0.10, 0.25],
                            [0.05, 0.40, 0.05]])
print("weather/commute independent:", independent(weather_commute))

# Build the 6x6 joint for two fair dice and test it.
# Then show Cov(X, X**2) = 0 for X uniform on {-1, 0, 1}, while Y = X**2
# is still a deterministic function of X.

import numpy as np

def independent(joint):
    px, py = joint.sum(axis=1), joint.sum(axis=0)
    return bool(np.allclose(joint, np.outer(px, py)))

weather_commute = np.array([[0.15, 0.10, 0.25],
                            [0.05, 0.40, 0.05]])
print("weather/commute independent:", independent(weather_commute))

dice = np.outer(np.full(6, 1 / 6), np.full(6, 1 / 6))
print("two dice independent:", independent(dice))

x = np.array([-1, 0, 1])
p = np.full(3, 1 / 3)
y = x**2
cov = (x * y * p).sum() - (x * p).sum() * (y * p).sum()
print("Cov(X, X^2) =", round(cov, 10))
print("but Y is determined by X:", bool(np.all(y == x**2)))

Independence means the joint factors into its marginals — for every pair of values. It licenses multiplying probabilities, factoring E[XY], and adding variances. Uncorrelated is strictly weaker: covariance only sees linear association, so a perfect parabolic relationship can hide inside a correlation of zero.

Next: covariance and correlation properly — how to measure the association that independence rules out.

16. Independence of random variables