Backpropagation: gradients by the chain rule

Nudging each weight in turn would be far too slow

Gradient descent needs, for every weight, how much the loss changes when that weight changes a little. One way to find out: nudge one weight, run the network again, and see how the loss moves. That costs one extra run of the network per weight. A network with a million weights would need a million runs for a single training step.

Backpropagation gets the same numbers from one forward run and one backward run.

A network is a chain of small steps, and slopes multiply along a chain

Take the smallest MLP: one input, one hidden unit with ReLU, one output, and a squared error. Written as a chain of steps:

z = w₁x + b₁  →  h = ReLU(z)  →  y = w₂h + b₂  →  L = (y − t)²

Here x is the input, t the target, and w₁, b₁, w₂, b₂ the four weights. Write ∂L/∂z for "how much L changes per unit change in z, with everything before z held fixed". The chain rule says that slopes along a chain multiply:

∂L/∂z = ∂L/∂y × ∂y/∂h × ∂h/∂z

Backpropagation evaluates this product from the left. It starts with ∂L/∂L = 1 at the loss. Each step back multiplies by one more local slope. So every gradient reuses the one computed just after it, and no work is repeated.

Try it. The figure shows the forward pass with x = 2, t = 2, w₁ = 0.5, b₁ = 0, w₂ = 3 and b₂ = 1. Before each press, predict the next orange number. Your goal is to reach ∂L/∂w₁.

Each kind of step has a short backward rule

Real networks are built from a few kinds of step. Each has a rule that turns the gradient arriving from the right into gradients for its inputs and its weights.

A whole linear layer applies the first two rules to every weight at once, as matrix arithmetic. The multiply rule also shows why the forward pass must keep its numbers: the gradient for w needs h. These stored values often take a large share of the memory that training uses, alongside the weights, their gradients and the optimizer's state.

backprop_torch.py

import torch

x, t = torch.tensor(2.0), torch.tensor(2.0)    # input and target
w1 = torch.tensor(0.5, requires_grad=True)
b1 = torch.tensor(0.0, requires_grad=True)
w2 = torch.tensor(3.0, requires_grad=True)
b2 = torch.tensor(1.0, requires_grad=True)

h = torch.relu(w1 * x + b1)   # forward pass
y = w2 * h + b2
loss = (y - t) ** 2
loss.backward()               # backward pass

print(loss.item())
print(w1.grad.item(), b1.grad.item(), w2.grad.item(), b2.grad.item())

4.0
24.0 12.0 4.0 4.0

The slow way, nudging one weight, agrees:

nudge_check.py

def loss(w1, b1=0.0, w2=3.0, b2=1.0, x=2.0, t=2.0):
    h = max(0.0, w1 * x + b1)
    y = w2 * h + b2
    return (y - t) ** 2

d = 0.001
print((loss(0.5 + d) - loss(0.5)) / d)   # nudge w1 and watch the loss

24.036000000000612

The nudge gives 24.036 rather than 24 because a nudge of 0.001 is small but not zero.