Automatic differentiation: forward and reverse mode · Advanced Calculus Topics

§14.3 computed a gradient by hand, chaining local derivatives together in a specific order. Automatic differentiation (autodiff) is the general algorithmic machinery that performs exactly that chaining automatically, for any computation built from elementary operations — and it produces exact derivatives, not the approximations of §3.0's difference quotient or the sometimes-unwieldy symbolic expressions SymPy has been building throughout this course. This lesson builds the two modes autodiff comes in, starting from a genuinely elegant piece of algebra: numbers with a built-in derivative attached.

Dual numbers: forward mode's core trick

Define a dual number as a+b\varepsilon, where \varepsilon is a symbol satisfying \varepsilon^2=0 (but \varepsilon\ne0 itself) — a purely formal device, not a real or complex number. Arithmetic on dual numbers automatically implements the derivative rules from Module 2:

$(a+b\varepsilon)+(c+d\varepsilon)=(a+c)+(b+d)\varepsilon$

— the sum rule (§2.3), since adding the \varepsilon-coefficients sums the derivatives.

$(a+b\varepsilon)(c+d\varepsilon)=ac+(ad+bc)\varepsilon+bd\varepsilon^2=ac+(ad+bc)\varepsilon$

(using \varepsilon^2=0 to drop the last term) — exactly the product rule (§2.3), \frac d{dx}(uv)=u'v+uv', reading a,c as values and b,d as their derivatives.

The pattern: represent x as the dual number x+1\cdot\varepsilon (value x, "seed" derivative 1, since \frac{dx}{dx}=1), run it through any sequence of arithmetic operations, and the \varepsilon- coefficient of the result is automatically the derivative of the whole computation with respect to x — with no symbolic expression ever built or simplified, just ordinary arithmetic carrying an extra number along for the ride.

Forward mode

Propagating dual numbers forward through a computation — value and derivative computed together, operation by operation, from input to output — is forward-mode automatic differentiation. It computes \dfrac{\partial(\text{output})}{\partial(\text{one chosen input})} in a single pass. Efficient when there are few inputs and many outputs — each pass handles one input variable's sensitivity across every output at once.

Reverse mode

Reverse-mode automatic differentiation instead runs the computation forward once (recording every intermediate value, building a computational graph), then propagates derivatives backward from the single final output to every input — exactly §14.3's backpropagation, now named as the specific case of reverse-mode autodiff applied to a neural network's loss. Efficient when there are many inputs and few outputs — a single backward pass computes the gradient with respect to every input parameter simultaneously, which is exactly the situation in machine learning: millions of weights (many inputs), one scalar loss (one output). This asymmetry is why neural networks are trained with reverse mode specifically, never forward mode — computing a million-parameter gradient via forward mode would need a million separate passes, one per parameter, while reverse mode gets all of them in one.

Doing it in Python

Implementing dual numbers from scratch, and using forward-mode autodiff to differentiate f(x)=x^2+3x at x=2 — comparing against the exact derivative f'(x)=2x+3:

class Dual:
    def __init__(self, val, deriv=0.0):
        self.val = val
        self.deriv = deriv

    def __add__(self, other):
        other = other if isinstance(other, Dual) else Dual(other, 0.0)
        return Dual(self.val + other.val, self.deriv + other.deriv)
    __radd__ = __add__

    def __mul__(self, other):
        other = other if isinstance(other, Dual) else Dual(other, 0.0)
        return Dual(self.val * other.val, self.deriv * other.val + self.val * other.deriv)
    __rmul__ = __mul__

    def __repr__(self):
        return f"Dual(value={self.val}, derivative={self.deriv})"

x = Dual(2, 1.0)          # seed: value=2, d(x)/dx=1
f = x * x + 3 * x         # computes f(x) = x^2 + 3x automatically
print(f)
print(f"exact check: f'(x) = 2x+3, f'(2) = {2*2+3}")

Extending the dual-number class to handle division, confirming a more complex function's derivative matches SymPy's symbolic answer:

class Dual:
    def __init__(self, val, deriv=0.0):
        self.val, self.deriv = val, deriv
    def __add__(self, o):
        o = o if isinstance(o, Dual) else Dual(o, 0.0)
        return Dual(self.val + o.val, self.deriv + o.deriv)
    __radd__ = __add__
    def __mul__(self, o):
        o = o if isinstance(o, Dual) else Dual(o, 0.0)
        return Dual(self.val * o.val, self.deriv * o.val + self.val * o.deriv)
    __rmul__ = __mul__
    def __pow__(self, n):
        return Dual(self.val**n, n * self.val**(n-1) * self.deriv)

x = Dual(3, 1.0)
f = x**3 + 2 * x * x   # f(x) = x^3 + 2x^2
print(f"autodiff: value={f.val}, derivative={f.deriv}")

import sympy as sp
xs = sp.Symbol('x')
fs = xs**3 + 2*xs**2
print(f"sympy check: f'(3) = {sp.diff(fs, xs).subs(xs, 3)}")

Confirming forward mode's inefficiency for many inputs versus reverse mode's single-pass advantage, conceptually:

n_inputs = 1_000_000
n_outputs = 1

print(f"forward mode: {n_inputs} passes needed (one per input)")
print(f"reverse mode: {n_outputs} pass needed (one per output)")
print("\nthis asymmetry is why neural networks train with reverse mode (backprop)")

Worked example

Use forward-mode automatic differentiation (dual numbers) to compute f'(2) for f(x)=x^2+3x.

Represent x=2 as the dual number 2+1\varepsilon (seed derivative 1, since \frac{dx}{dx}=1).

Compute x^2=x\cdot x, using the dual-number multiplication rule:

$(2+1\varepsilon)(2+1\varepsilon)=2\cdot2+(2\cdot1+1\cdot2)\varepsilon=4+4\varepsilon$

Compute 3x:

$3\cdot(2+1\varepsilon)=6+3\varepsilon$

(multiplying a dual number by an ordinary constant 3 — equivalently, 3+0\varepsilon — gives 3\cdot2+ (3\cdot1+0\cdot2)\varepsilon=6+3\varepsilon.)

Add:

$(4+4\varepsilon)+(6+3\varepsilon)=10+7\varepsilon$

$\boxed{f(2)=10,\qquad f'(2)=7}$

Sanity check. Directly: f'(x)=2x+3, so f'(2)=4+3=7 — matching the \varepsilon-coefficient exactly. And f(2)=4+6=10 matches the dual number's ordinary (non-\varepsilon) part too. Every step used nothing but ordinary multiplication and addition — no symbolic expression for f'(x) was ever built or simplified, and no finite difference (§3.0) approximated anything; the exact derivative fell out purely from tracking the \varepsilon-coefficient through ordinary arithmetic. ✓

Your turn

1. Using dual numbers by hand, compute f(3) and f'(3) for f(x)=x^2 (represent x as 3+1\varepsilon and multiply it by itself).

2. Explain why a neural network with 10 million parameters and a single scalar loss is trained using reverse-mode (not forward-mode) autodiff.

3. True or false: automatic differentiation produces an approximation to the true derivative, in the same sense that §3.0's difference quotient with a small but nonzero h does.

Solutions

1. x=3+1\varepsilon.

$x\cdot x=(3+1\varepsilon)(3+1\varepsilon)=9+(3+3)\varepsilon=9+6\varepsilon$

$\boxed{f(3)=9,\qquad f'(3)=6}$

(Check: f'(x)=2x\Rightarrow f'(3)=6 ✓.)

2. The concept section's asymmetry: forward mode needs one full pass per input to get that input's contribution to the output, meaning 10 million passes for 10 million parameters. Reverse mode needs one pass per output, and there's only one output (the scalar loss) — so a single backward pass computes the gradient with respect to all 10 million parameters simultaneously. Reverse mode is astronomically cheaper whenever inputs vastly outnumber outputs, which is always the case in neural network training.

3. False. Dual-number arithmetic computes the exact derivative at every step, using algebraic identities (\varepsilon^2=0) rather than a limiting process with a small but nonzero step size. This is the key distinction from §3.0's difference quotient (which genuinely approximates, with error shrinking only as h\to0) — automatic differentiation has no such error term at all; the worked example's f'(2)=7 is exact, not an approximation converging toward 7.

Check yourself in code

Using dual numbers, compute f(2) and f'(2) for f(x)=x^2+3x via forward-mode automatic differentiation.

Print exactly this:

Dual(value=10, derivative=7.0)

class Dual:
    def __init__(self, val, deriv=0.0):
        self.val = val
        self.deriv = deriv

    def __add__(self, other):
        other = other if isinstance(other, Dual) else Dual(other, 0.0)
        return Dual(self.val + other.val, self.deriv + other.deriv)
    __radd__ = __add__

    def __mul__(self, other):
        other = other if isinstance(other, Dual) else Dual(other, 0.0)
        return Dual(self.val * other.val, self.deriv * other.val + self.val * other.deriv)
    __rmul__ = __mul__

    def __repr__(self):
        return f"Dual(value={self.val}, derivative={self.deriv})"

x = Dual(2, 1.0)
f = x * x + 3 * x
print(f)

class Dual:
    def __init__(self, val, deriv=0.0):
        self.val = val
        self.deriv = deriv

    def __add__(self, other):
        other = other if isinstance(other, Dual) else Dual(other, 0.0)
        return Dual(self.val + other.val, self.deriv + other.deriv)
    __radd__ = __add__

    def __mul__(self, other):
        other = other if isinstance(other, Dual) else Dual(other, 0.0)
        return Dual(self.val * other.val, self.deriv * other.val + self.val * other.deriv)
    __rmul__ = __mul__

    def __repr__(self):
        return f"Dual(value={self.val}, derivative={self.deriv})"

x = Dual(2, 1.0)
f = x * x + 3 * x
print(f)

Dual numbers, a+b\varepsilon with \varepsilon^2=0, implement Module 2's sum and product rules through pure algebra — running a computation with a "seeded" dual input carries the exact derivative along automatically, with no symbolic expression ever assembled. Forward mode propagates these dual numbers input-to-output and suits few-input, many-output problems; reverse mode — exactly §14.3's backpropagation — runs forward once and then propagates derivatives backward, making it the only practical choice whenever a computation has vastly more inputs (millions of weights) than outputs (one loss), which is always the situation training a neural network.

Next: putting every tool from this module to direct use, deriving the matrix-calculus solutions behind linear and logistic regression.

15. Automatic differentiation: forward and reverse mode