Concept units

The chain rule: slopes of chained steps

Basic

Read these first if they are new to you

The idea in one paragraph

Many calculations are a chain of steps: the output of one step is the input of the next. The chain rule says how fast the final output changes when you nudge the first input. You find each step's slope at the point where the chain passes through it, and multiply the slopes together. If step one triples a small change and step two doubles it, the whole chain multiplies it by six.

Rates of change multiply along a chain

Think of three gears in a row. Gear B turns 2 times for each turn of gear A. Gear C turns 3 times for each turn of gear B. So gear C turns 2 × 3 = 6 times for each turn of gear A. The rates multiply.

A calculation works the same way. Take y = (3x + 1)2. It is two steps. First u = 3x + 1, then y = u2. Nudge x a little and u moves 3 times as much. Nudge u a little and y moves 2u times as much, because the slope of u2 is 2u, as the derivative unit shows. So y moves 3 × 2u times as much as x.

This is the chain rule. In symbols, with each fraction meaning "a tiny change on top per tiny change below":

dy/dx = (dy/du) × (du/dx)

A small step in x is stretched by each step of the chain in turn
A step of 0.1 in x becomes 0.3 in u (× 3), then 2.49 in y (× 8.3).
Chain rule: 2u × 3 = 2 × 4 × 3 = 24
Try this. The blue bar is a step of 0.1 in x. The orange bar is where that step lands on the u line, and the green bar where it lands on the y line. At x = 1, the measured stretch of 2.49 / 0.1 ≈ 25 is close to the chain rule's 24, and it gets closer for smaller steps. Goal: predict the x at which the green bar is shortest, then slide to check.
Worked example

Find the slope of y = (3x + 1)2 at x = 1.

  1. Run the chain forward: u = 3 × 1 + 1 = 4, and y = 42 = 16.
  2. Slope of the first step: du/dx = 3, the same everywhere.
  3. Slope of the second step at u = 4: dy/du = 2 × 4 = 8.
  4. Multiply: dy/dx = 8 × 3 = 24.
  5. Check with a small step: at x = 1.001, y = 4.0032 = 16.024009. The change 0.024009 divided by 0.001 is about 24.

Your turn. Find the slope of y = (2x − 5)2 at x = 4.

Answer

u = 2 × 4 − 5 = 3. The slopes are du/dx = 2 and dy/du = 2 × 3 = 6. Multiply: 12.

Longer chains multiply more slopes

A chain of four steps has four slopes, and the chain rule multiplies all four. The work has two passes. A forward pass runs the chain and records the value at each step. A backward pass starts at the output and multiplies the slopes back toward the input, using those recorded values. Training a neural network does this for millions of adjustable numbers, called weights, at once. The automatic-gradients unit shows how a program keeps track.

This is a simplification

Every step here has one input and one output. When a step has several inputs, or one value feeds several later steps, the chain rule adds up the products along every path from input to output. The gradient unit covers functions with several inputs. The rule also needs each step to have a slope at the point the chain passes through.

chain.py: the chain rule against a direct small-step estimate

def inner(x):
    return 3 * x + 1      # slope 3 everywhere

def outer(u):
    return u ** 2         # slope 2u

x = 1.0
u = inner(x)
print(u, 2 * u * 3)       # chain rule: outer slope times inner slope

h = 0.001
print(round((outer(inner(x + h)) - outer(inner(x))) / h, 3))   # check with a small step

4.0 24.0
24.009

Check yourself

  1. A recipe step doubles a quantity, the next step adds 7, and the last step multiplies by 5. If the input grows by 1, how much does the output grow?

    Answer

    By 10. The slopes are 2, 1 and 5, and 2 × 1 × 5 = 10. Adding a constant has slope 1, so it does not change the rate.

  2. For y = (3x + 1)2, at which x is the slope zero? Why?

    Answer

    At x = −1/3, where u = 0. The second step's slope 2u is zero there, and anything times zero is zero.

  3. In a chain of three steps, the slopes at the current point are 0.5, 4 and −2. What is the slope of the whole chain?

    Answer

    0.5 × 4 × (−2) = −4.

Go further

Optional background, for readers who want more than this page.

Where this shows up

Units that build on this one