14. The chain rule as backpropagation

A neural network is, underneath its name, a composition of functions — input passed through one layer, then the next, then the next, until a final loss measures how wrong the output is. Training the network means computing how that loss changes with respect to every internal parameter, so gradient descent (§14.0) knows which way to step. Backpropagation is simply §10.4's multivariable chain rule, applied systematically to a long composition, computed in a specific order that avoids massive redundant work.

A minimal network

Consider the smallest possible example: input x, one weight w, one bias b, a nonlinear activation function \sigma (the sigmoid, \sigma(z)=\frac1{1+e^{-z}}), and a loss comparing the output to a target y:

z=wx+b\qquad a=\sigma(z)\qquad L=\frac12(a-y)^2

Three composed steps: L depends on a, which depends on z, which depends on w and b. Training needs \dfrac{\partial L}{\partial w} and \dfrac{\partial L}{\partial b} — how a small nudge to each parameter changes the final loss.

Applying the chain rule, one step at a time

\frac{\partial L}{\partial w}=\frac{\partial L}{\partial a}\cdot\frac{\partial a}{\partial z}\cdot\frac{\partial z}{\partial w}

— exactly §10.4's chain rule, with three intermediate variables chained together instead of two. Each factor is a simple, local derivative:

\frac{\partial L}{\partial a}=a-y\qquad\frac{\partial a}{\partial z}=\sigma(z)\big(1-\sigma(z)\big)=a(1-a)\qquad\frac{\partial z}{\partial w}=x

(the middle factor is the sigmoid's own derivative, derivable directly from §2.6's exponential and the quotient rule, §2.3). Multiplying:

\frac{\partial L}{\partial w}=(a-y)\cdot a(1-a)\cdot x

By the identical chain, but with \frac{\partial z}{\partial b}=1 instead of x:

\frac{\partial L}{\partial b}=(a-y)\cdot a(1-a)\cdot1

Why "backward" propagation

Notice \dfrac{\partial L}{\partial b} reuses the exact same first two factors, (a-y)\cdot a(1-a), that \dfrac{\partial L}{\partial w} needed. Computing that shared piece once, then multiplying it by each parameter's own local factor (x for w, 1 for b), avoids recomputing it twice. In a real network with millions of parameters spread across many layers, this reuse is not a minor optimization — it's the entire reason training is computationally feasible at all.

Backpropagation names this specific order of computation: start from the loss L (the very end of the composition) and multiply chain-rule factors moving backward through the network, layer by layer, always reusing the "how much does the loss change per unit change in this layer's output" quantity computed one step previously, rather than starting fresh from each parameter and working forward. §14.4 makes this "reuse intermediate results, working backward" strategy precise, under the name reverse-mode automatic differentiation — backpropagation is that general algorithm, specialized to neural networks.

Doing it in Python

Computing \dfrac{\partial L}{\partial w} and $\dfrac{\partial L}{\partial b}$ two ways — via the chain rule, step by step, and via direct symbolic differentiation of the fully composed expression — confirming they agree:

import sympy as sp

w, b, x, y = sp.symbols('w b x y')
z = w*x + b
a = 1 / (1 + sp.exp(-z))
L = (a - y)**2 / 2

# direct differentiation of the fully composed expression
dL_dw_direct = sp.diff(L, w)
dL_db_direct = sp.diff(L, b)

# chain rule, step by step
dL_da = a - y
da_dz = a * (1 - a)
dz_dw = x
dz_db = 1

dL_dw_chain = dL_da * da_dz * dz_dw
dL_db_chain = dL_da * da_dz * dz_db

print(f"dL/dw match: {sp.simplify(dL_dw_direct - dL_dw_chain) == 0}")
print(f"dL/db match: {sp.simplify(dL_db_direct - dL_db_chain) == 0}")

Evaluating the chain-rule computation numerically for specific values, and confirming the shared factor is genuinely reused:

def sigmoid(z):
    return 1 / (1 + 2.718281828459045**(-z))

x_val, w_val, b_val, y_val = 2, 0.5, 0.1, 1

z_val = w_val * x_val + b_val
a_val = sigmoid(z_val)
L_val = 0.5 * (a_val - y_val)**2

shared_factor = (a_val - y_val) * a_val * (1 - a_val)   # dL/da * da/dz, computed once
dL_dw = shared_factor * x_val
dL_db = shared_factor * 1

print(f"z = {z_val:.6f}")
print(f"a = {a_val:.6f}")
print(f"L = {L_val:.6f}")
print(f"shared factor (dL/da * da/dz) = {shared_factor:.6f}")
print(f"dL/dw = shared_factor * x = {dL_dw:.6f}")
print(f"dL/db = shared_factor * 1 = {dL_db:.6f}")

Extending the chain by one more layer — confirming the pattern scales, using another weight v feeding into z through a second linear transformation:

import sympy as sp

u, v, x, y = sp.symbols('u v x y')
h = v * x          # a first linear layer
z = u * h          # a second linear layer feeding into the same sigmoid+loss as before
a = 1 / (1 + sp.exp(-z))
L = (a - y)**2 / 2

# chain rule through three layers: L -> a -> z -> h -> v
dL_da = a - y
da_dz = a * (1 - a)
dz_dh = u
dh_dv = x

dL_dv_chain = dL_da * da_dz * dz_dh * dh_dv
dL_dv_direct = sp.diff(L, v)
print(f"three-layer chain rule match: {sp.simplify(dL_dv_chain - dL_dv_direct) == 0}")

Worked example

For z=wx+b, a=\sigma(z), L=\frac12(a-y)^2, with x=2, w=0.5, b=0.1, y=1, compute \dfrac{\partial L}{\partial w} and \dfrac{\partial L}{\partial b}.

z=(0.5)(2)+0.1=1.1

a=\sigma(1.1)=\frac1{1+e^{-1.1}}\approx0.7503

Chain-rule factors:

\frac{\partial L}{\partial a}=a-y=0.7503-1=-0.2497

\frac{\partial a}{\partial z}=a(1-a)=0.7503(0.2497)\approx0.1874

\text{shared factor}=(-0.2497)(0.1874)\approx-0.04679

\frac{\partial L}{\partial w}=(-0.04679)\cdot x=(-0.04679)(2)\approx\boxed{-0.09359}

\frac{\partial L}{\partial b}=(-0.04679)\cdot1\approx\boxed{-0.04679}

Sanity check. Both gradients are negative, meaning gradient descent (§14.0) would increase w and b to reduce the loss — sensible, since the target y=1 exceeds the current output a\approx0.75, so pushing a higher (by increasing z=wx+b, which increasing either w or b does) moves the output toward the target and should indeed lower the loss. And \dfrac{\partial L}{\partial w} is exactly double \dfrac{\partial L}{\partial b} — matching $\dfrac{\partial z}{\partial w}=x=2$ versus \dfrac{\partial z}{\partial b}=1 precisely, since both gradients share the identical upstream factor and differ only by this one local multiplier. ✓

Your turn

1. Using the shared factor $\frac{\partial L}{\partial a}\cdot \frac{\partial a}{\partial z}\approx-0.04679$ from the worked example, find \dfrac{\partial L}{\partial x} (treating x as if it too were a trainable parameter, with \dfrac{\partial z}{\partial x}=w).

2. Explain, in one sentence, why computing the shared factor \frac{\partial L}{\partial a}\cdot\frac{\partial a}{\partial z} once and reusing it for both \dfrac{\partial L}{\partial w} and \dfrac{\partial L}{\partial b} saves computation compared to computing each gradient from scratch.

3. True or false: backpropagation requires a fundamentally different mathematical rule than the chain rule already covered in Module 10.

Solutions

1. \dfrac{\partial z}{\partial x}=w=0.5.

\frac{\partial L}{\partial x}=(-0.04679)(0.5)\approx\boxed{-0.0234}

2. Both gradients need the identical upstream quantity \frac{\partial L}{\partial a}\cdot\frac{\partial a}{\partial z} (how sensitive the loss is to the pre-activation value z); computing it once and multiplying by each parameter's own local derivative (\frac{\partial z}{\partial w} or \frac{\partial z}{\partial b}) avoids redoing that shared, often expensive part of the computation for every single parameter.

3. False. Every computation in this lesson was §10.4's multivariable chain rule, applied repeatedly to a composed function — nothing new was introduced beyond that single rule and a disciplined order of evaluation (computing shared upstream factors once, moving backward through the composition). Backpropagation's contribution is algorithmic — an efficient order to apply a rule already fully established — not a new piece of calculus.

Check yourself in code

For z=wx+b, a=\sigma(z), L=\frac12(a-y)^2 with x=2, w=0.5, b=0.1, y=1, compute z, a, and the gradients $\dfrac{\partial L}{\partial w}$ and \dfrac{\partial L}{\partial b} via the chain rule.

Print exactly this:

z = 1.100000
a = 0.750260
dL/dw = -0.093587
dL/db = -0.046794

def sigmoid(z):
    return 1 / (1 + 2.718281828459045**(-z))

x_val, w_val, b_val, y_val = 2, 0.5, 0.1, 1

z_val = w_val * x_val + b_val
print(f"z = {z_val:.6f}")

a_val = sigmoid(z_val)
print(f"a = {a_val:.6f}")

shared_factor = (a_val - y_val) * a_val * (1 - a_val)
dL_dw = shared_factor * x_val
dL_db = shared_factor * 1
print(f"dL/dw = {dL_dw:.6f}")
print(f"dL/db = {dL_db:.6f}")

def sigmoid(z):
    return 1 / (1 + 2.718281828459045**(-z))

x_val, w_val, b_val, y_val = 2, 0.5, 0.1, 1

z_val = w_val * x_val + b_val
print(f"z = {z_val:.6f}")

a_val = sigmoid(z_val)
print(f"a = {a_val:.6f}")

shared_factor = (a_val - y_val) * a_val * (1 - a_val)
dL_dw = shared_factor * x_val
dL_db = shared_factor * 1
print(f"dL/dw = {dL_dw:.6f}")
print(f"dL/db = {dL_db:.6f}")

Backpropagation is §10.4's multivariable chain rule, applied to a network's full composition of layers and evaluated in a specific order — starting from the loss and working backward — that computes and reuses each shared upstream factor exactly once rather than recomputing it separately for every parameter. Every gradient in this lesson's minimal network came from multiplying three ordinary local derivatives together; nothing beyond the chain rule itself was needed, only discipline about the order of multiplication.

Next: automatic differentiation — the general algorithmic framework, in both forward and reverse mode, that makes this backward chain-rule computation happen automatically inside every modern deep learning framework.