Hyperbolic functions, and why tanh shows up in neural nets · Calculus I: Limits & Derivatives

These are built from exponentials, they satisfy identities that mirror the trigonometric ones with a sign flipped, and their derivatives are tidier than anything else in this module.

$\sinh x = \frac{e^x - e^{-x}}{2}, \qquad \cosh x = \frac{e^x + e^{-x}}{2}, \qquad \tanh x = \frac{\sinh x}{\cosh x} = \frac{e^x - e^{-x}}{e^x + e^{-x}}$

Pronounced "sinch", "cosh", "tanch". Note \cosh is the even part of e^x and \sinh the odd part, so e^x = \cosh x + \sinh x — the decomposition from §0.2 applied to the exponential.

Why "hyperbolic"

The Pythagorean identity has a sign changed:

$\cosh^2 x - \sinh^2 x = 1$

Verify it directly:

$\left(\frac{e^x+e^{-x}}{2}\right)^2 - \left(\frac{e^x-e^{-x}}{2}\right)^2 = \frac{(e^{2x} + 2 + e^{-2x}) - (e^{2x} - 2 + e^{-2x})}{4} = \frac44 = 1$

Compare the two:

$\cos^2t + \sin^2t = 1 \quad\text{parameterises the circle } x^2+y^2=1$ $\cosh^2t - \sinh^2t = 1 \quad\text{parameterises the hyperbola } x^2-y^2=1$

Hence the name. The parameter t isn't an angle in the hyperbolic case — it's twice the area of the sector swept out, which is also true of the circular case (area = \frac{t}{2} for the unit circle, from §0.1) but less often noticed.

The derivatives

$\frac{d}{dx}\sinh x = \cosh x, \qquad \frac{d}{dx}\cosh x = \sinh x$

No minus sign. That's the one difference from the trigonometric pair, and it follows instantly from the definitions:

$\frac{d}{dx}\frac{e^x - e^{-x}}{2} = \frac{e^x + e^{-x}}{2} = \cosh x$

The chain rule supplies -1 from e^{-x} each time, and in \cosh's derivative it turns +e^{-x} into -e^{-x}, producing \sinh.

For tangent, the quotient rule with the hyperbolic identity:

$\frac{d}{dx}\tanh x = \frac{\cosh^2x - \sinh^2x}{\cosh^2 x} = \frac{1}{\cosh^2x} = \operatorname{sech}^2 x = 1 - \tanh^2 x$

That last form is the one to remember: \tanh' = 1 - \tanh^2. The derivative is a function of the output alone, so if you already have \tanh x you get its derivative for free with no new transcendental evaluation. That property is worth real money, as the next section explains.

f(x)	f'(x)
\sinh x	\cosh x
\cosh x	\sinh x
\tanh x	\operatorname{sech}^2x = 1-\tanh^2x
\operatorname{sech} x	-\operatorname{sech}x\tanh x

The second-derivative fact mirrors §2.5's with a sign change:

$\frac{d^2}{dx^2}\cosh x = \cosh x \implies y'' = +y$

Trigonometric functions solve y'' = -y (oscillation); hyperbolic ones solve y'' = +y (exponential growth and decay). Same equation, opposite sign, entirely different physics — and §13.5 shows exactly where the sign comes from.

\tanh as an activation function

\tanh squashes all of \mathbb R into (-1,1), smoothly, monotonically, and with \tanh(0) = 0 and \tanh'(0) = 1. That combination made it the standard neural-network activation before ReLU, and it's still ubiquitous in recurrent architectures (the LSTM gate is \tanh and \sigma).

The reason it was practical is the derivative identity. Training needs \tanh'(x) at every node on every backward pass (§14.3). Because

$\tanh'(x) = 1 - \tanh(x)^2$

and the forward pass already computed \tanh(x), the backward pass is one multiply and one subtract — no exponentials at all. The logistic sigmoid \sigma(x) = \frac{1}{1+e^{-x}} has the same property, \sigma' = \sigma(1-\sigma), and they're relatives:

$\tanh x = 2\sigma(2x) - 1$

Where it goes wrong is also a derivative fact. For |x| \gtrsim 3, \tanh' \approx 0, so gradients passing through a saturated unit are multiplied by nearly zero. Stack a few layers of that and the chain rule's product (§2.4) underflows — the vanishing gradient problem, which is precisely why ReLU (derivative exactly 1 on the positive side) displaced \tanh in deep networks.

An entire architectural shift in machine learning, decided by the shape of one derivative.

The catenary

Hang a chain between two posts. It does not form a parabola. It forms

$y = a\cosh\!\left(\frac{x}{a}\right)$

the catenary, and this is the curve's defining property: it's the shape a uniform flexible cable takes under its own weight. Galileo guessed parabola; Huygens, Leibniz and Johann Bernoulli got it right in 1691.

The reason \cosh appears is that the equilibrium condition works out to y'' = \frac1a\sqrt{1+(y')^2}, and \cosh is the solution — a §13 problem you'll be able to solve. Inverted, the same curve is the optimal arch, which is why the Gateway Arch in St. Louis is a (weighted) catenary.

The inverses

$\frac{d}{dx}\sinh^{-1}x = \frac{1}{\sqrt{x^2+1}}, \qquad \frac{d}{dx}\tanh^{-1}x = \frac{1}{1-x^2} \quad (|x|<1)$

Compare with \arcsin' = \frac{1}{\sqrt{1-x^2}} and \arctan' = \frac{1}{1+x^2} — signs flipped again. And because these are built from exponentials, the inverses have closed forms that the trigonometric ones don't:

$\sinh^{-1}x = \ln\!\left(x + \sqrt{x^2+1}\right)$

Both of these turn up as integration results in §4.8, which is the main reason a calculus course covers hyperbolics at all.

Doing it in Python

The identity and the derivatives:

from math import sinh, cosh, tanh, exp

def numerical(f, x, h=1e-6):
    return (f(x + h) - f(x - h)) / (2 * h)

print(f"{'x':>6} {'cosh^2-sinh^2':>16} {'d(sinh)':>12} {'cosh':>12} "
      f"{'d(cosh)':>12} {'sinh':>12}")
for x in (0.0, 0.5, 1.0, 2.0):
    print(f"{x:>6} {cosh(x)**2 - sinh(x)**2:>16.10f} "
          f"{numerical(sinh, x):>12.6f} {cosh(x):>12.6f} "
          f"{numerical(cosh, x):>12.6f} {sinh(x):>12.6f}")

print("\nthe identity is exactly 1 everywhere, and neither derivative has a minus sign")

The activation-function property, and the cost it saves:

from math import tanh, cosh

def numerical(f, x, h=1e-6):
    return (f(x + h) - f(x - h)) / (2 * h)

print(f"{'x':>6} {'tanh(x)':>12} {'1 - tanh^2':>14} {'sech^2':>12} {'numerical':>12}")
for x in (0.0, 0.5, 1.0, 2.0, 3.0):
    t = tanh(x)
    print(f"{x:>6} {t:>12.8f} {1 - t*t:>14.8f} {1/cosh(x)**2:>12.8f} "
          f"{numerical(tanh, x):>12.8f}")

print("\nthe derivative is a function of the OUTPUT: no new exp() needed on the backward pass")

Saturation, and why deep \tanh networks stopped training:

from math import tanh

print(f"{'x':>6} {'tanh(x)':>12} {'derivative':>12}")
for x in (0, 1, 2, 3, 5, 8):
    print(f"{x:>6} {tanh(x):>12.8f} {1 - tanh(x)**2:>12.2e}")

print("\ngradient through 10 saturated layers, chain rule (2.4) multiplying:")
for x in (0.5, 1.0, 2.0, 3.0):
    g = (1 - tanh(x)**2) ** 10
    print(f"  units at x={x}: {g:.3e}")

print("\nat x=3 the signal is gone. that is the vanishing gradient problem,")
print("and it is why ReLU (derivative exactly 1) replaced tanh in deep nets.")

The catenary is not a parabola:

from math import cosh

a = 1.0
print(f"{'x':>6} {'catenary a*cosh(x/a)':>22} {'parabola 1 + x^2/2':>20} {'gap':>12}")
for x in (0.0, 0.5, 1.0, 2.0, 3.0):
    cat = a * cosh(x / a)
    par = 1 + x*x / 2
    print(f"{x:>6} {cat:>22.8f} {par:>20.8f} {cat - par:>12.6f}")

print("\nthey agree to second order and diverge after -- Galileo's guess was")
print("a very good approximation for a shallow sag, and wrong for a deep one")

Worked example

Differentiate y = \tanh(x^2) and find where the slope is largest.

Chain rule with u = x^2:

$\frac{dy}{dx} = \operatorname{sech}^2(x^2)\cdot 2x = 2x\left[1 - \tanh^2(x^2)\right]$

Where is it largest? Two competing effects: 2x grows, and 1 - \tanh^2(x^2) decays fast once x^2 gets past about 2. So there's a maximum somewhere in between, and you'd find it by setting the second derivative to zero (§3.5) — numerically it lands at x \approx 0.7224, where the slope is about 1.113.

At x = 0: y' = 0, and indeed \tanh(x^2) \approx x^2 near the origin, which is flat there ✓.

A second one. \frac{d}{dx}\ln(\cosh x):

$= \frac{\sinh x}{\cosh x} = \tanh x$

So \ln\cosh x is an antiderivative of \tanh x. This function is the log-cosh loss in machine learning — quadratic for small errors, linear for large ones, and smooth everywhere, which makes it a differentiable stand-in for the absolute-value loss. Its gradient being exactly \tanh is why it's cheap.

Your turn

1. Show \sinh(2x) = 2\sinh x\cosh x from the definitions.

2. \dfrac{d}{dx}\cosh(3x^2)

3. \dfrac{d}{dx}\tanh(\ln x)

4. Show that y = A\cosh(kx) + B\sinh(kx) satisfies y'' = k^2 y.

Solutions

1. Expand the right side:

$2\cdot\frac{e^x-e^{-x}}{2}\cdot\frac{e^x+e^{-x}}{2} = \frac{(e^x-e^{-x})(e^x+e^{-x})}{2} = \frac{e^{2x} - e^{-2x}}{2}$

which is exactly \sinh(2x) ✓.

Note this is the same form as \sin 2x = 2\sin x\cos x — no sign change here. The general pattern (Osborn's rule) is that hyperbolic identities match trigonometric ones except that every product of two sines flips sign. There's no such product here, so the identity carries over unchanged.

2. Chain rule:

$\sinh(3x^2)\cdot 6x = \boxed{6x\sinh(3x^2)}$

3. Chain rule with u = \ln x:

$\operatorname{sech}^2(\ln x)\cdot\frac1x = \boxed{\frac{1 - \tanh^2(\ln x)}{x}}$

Worth simplifying: \tanh(\ln x) = \frac{x - 1/x}{x + 1/x} = \frac{x^2-1}{x^2+1}, so the whole function is rational, and its derivative is \frac{4x}{(x^2+1)^2}. A good reminder that hyperbolic functions of logarithms are secretly algebraic.

4. Differentiate twice, using \cosh' = \sinh and \sinh' = \cosh with the chain rule contributing a k each time:

$y' = Ak\sinh(kx) + Bk\cosh(kx)$

$y'' = Ak^2\cosh(kx) + Bk^2\sinh(kx) = k^2\left[A\cosh(kx) + B\sinh(kx)\right] = k^2y \qquad\blacksquare$

Contrast y = A\cos(kx)+B\sin(kx), which satisfies y'' = -k^2y. The sign of the constant decides whether solutions oscillate or grow exponentially — the central dichotomy of §13.5's second-order equations, and the reason a hanging chain and a vibrating string obey superficially similar equations with completely different answers.

Check yourself in code

Verify the hyperbolic identity and derivative rules.

For x \in \{0, 0.5, 1, 2\}, print \cosh^2x - \sinh^2x to 8 decimals, then the central difference of \sinh (h=10^{-6}) against \cosh x, both to 6 decimals. Finally print \tanh'(1) numerically against 1 - \tanh^2(1), to 8 decimals.

Print exactly this:

x=0.0   cosh^2-sinh^2=1.00000000  d(sinh)=1.000000 cosh=1.000000
x=0.5   cosh^2-sinh^2=1.00000000  d(sinh)=1.127626 cosh=1.127626
x=1.0   cosh^2-sinh^2=1.00000000  d(sinh)=1.543081 cosh=1.543081
x=2.0   cosh^2-sinh^2=1.00000000  d(sinh)=3.762196 cosh=3.762196
tanh'(1) numeric 0.41997434  1-tanh^2 0.41997434

from math import sinh, cosh, tanh

def numerical(f, x, h=1e-6):
    return (f(x + h) - f(x - h)) / (2 * h)

for x in (0.0, 0.5, 1.0, 2.0):
    # print the identity, then d(sinh) numerically against cosh
    print(f"x={x:<5} ...")

# then tanh'(1) two ways

from math import sinh, cosh, tanh

def numerical(f, x, h=1e-6):
    return (f(x + h) - f(x - h)) / (2 * h)

for x in (0.0, 0.5, 1.0, 2.0):
    print(f"x={x:<5} cosh^2-sinh^2={cosh(x)**2 - sinh(x)**2:.8f}  "
          f"d(sinh)={numerical(sinh, x):.6f} cosh={cosh(x):.6f}")

print(f"tanh'(1) numeric {numerical(tanh, 1.0):.8f}  1-tanh^2 {1 - tanh(1.0)**2:.8f}")

\sinh and \cosh are the odd and even parts of e^x, they satisfy \cosh^2 - \sinh^2 = 1 (a hyperbola, hence the name), and they differentiate into each other with no minus sign. \tanh' = 1 - \tanh^2 expresses the derivative in terms of the output, which is what made \tanh a cheap activation function — and its saturation is what made deep \tanh networks untrainable. \cosh is the shape of a hanging chain, and both families solve y'' = k^2y where the trigonometric ones solve y'' = -k^2y.

Next: differentiating curves that aren't functions at all.

24. Hyperbolic functions, and why tanh shows up in neural nets