Residual connections

Read these first if they are new to you

The multilayer perceptron (MLP) Intermediate
The chain rule: slopes of chained steps Basic

The idea in one paragraph

A residual connection adds a layer's input back to its output: y = x + F(x). The layer F then only has to learn a change to x, not a whole new vector. If F outputs zeros, x passes through unchanged. Deep networks built this way train well, because the input, and the training signal on its way back, can pass straight through the addition. Picture a running vector that each block edits a little.

The block adds its input to its output

Take any layer or small network F whose output has the same size as its input, such as an MLP. A residual block computes F(x) and adds x to it. The path that carries x around F is the skip path; people also call the whole thing a skip connection. F is the residual branch, because it computes the residual: the difference between the output and the input.

A residual block, and what it does to one vector

What to notice. Left: the input reaches the output twice, once through F and once along the skip path (blue). Right: F(x) is a short arrow added at the tip of x, so y stays close to x. The block edits x; it does not replace it.

Predict first. If every weight in F were 0, what would the block output?

Worked example: one residual block on x = [2, −1]

Let F be a two-layer MLP with a ReLU between the layers and no biases: F(x) = W₂ ReLU(W₁x), with W₁ = [[0.5, 0], [0, −0.5]] and W₂ = [[0.1, 0], [0.2, 0.2]] (each inner pair is one row).

W₁x = [0.5 × 2 + 0 × (−1), 0 × 2 + (−0.5) × (−1)] = [1.0, 0.5].
ReLU keeps positive numbers and turns negative ones into 0: [1.0, 0.5].
W₂ times that: [0.1 × 1.0 + 0 × 0.5, 0.2 × 1.0 + 0.2 × 0.5] = [0.1, 0.3]. This is F(x).
Add the input: y = [2, −1] + [0.1, 0.3] = [2.1, −0.7].

The output is the input plus a small edit. With all weights 0, F(x) = [0, 0] and the block outputs x itself.

The addition needs F(x) and x to have the same size, entry for entry. That is why blocks with residual connections never change the number of entries in a vector.

The skip path keeps the training signal alive through many layers

Training changes each weight according to how much the final error changes when that weight changes (see gradient descent). For a weight in an early layer, that effect must pass through every later layer. In one dimension, each layer multiplies it by the layer's slope, its derivative. This is the chain rule: the slope of a chain of steps is the product of the steps' slopes.

A plain layer y = f(x) has slope f′(x). A residual block y = x + f(x) has slope 1 + f′(x). The "1" comes from the skip path. The table multiplies the slopes of stacked layers:

Layers	Plain, slope 0.5 each	Residual, slope 1 + 0.05 each	Residual, slope 1 − 0.05 each
1	0.5	1.05	0.95
10	0.00098	1.63	0.60
20	0.00000095	2.65	0.36

Twenty plain layers shrink the signal to about one millionth, so the early layers barely learn. This is called a vanishing gradient. Twenty residual blocks with small edits keep it between about a third and three times its size. At the start of training the edits are small, so the signal reaches every layer.

This mattered in practice. In 2015, He and colleagues reported that a plain 56-layer image network had a higher training error than a plain 20-layer one, even though it could in principle copy the shallow network. With residual connections they trained networks of 152 layers. Today every transformer wraps each of its sublayers in a residual connection (see the transformer block).

residual.py

import numpy as np

x = np.array([2.0, -1.0])
W1 = np.array([[0.5, 0.0], [0.0, -0.5]])
W2 = np.array([[0.1, 0.0], [0.2, 0.2]])

F = W2 @ np.maximum(0, W1 @ x)    # the residual branch: a small MLP
print(F, x + F)

# Products of 20 slopes: plain layers, then residual blocks
print(0.5 ** 20, round(1.05 ** 20, 2), round(0.95 ** 20, 2))

[0.1 0.3] [ 2.1 -0.7]
9.5367431640625e-07 2.65 0.36

The one-dimensional story hides two details

Real layers take vectors, so each slope is a matrix of slopes, and the products are products of matrices. The skip path still contributes an identity matrix (the matrix version of "1") to each block. Researchers still study the other reasons residual networks train well. Also, when F must change the vector's size, the skip path carries Wx instead of x, with a learned matrix W that makes the sizes match.

Check yourself

A residual block receives x = [1, 1], and its branch gives F(x) = [0.2, −0.1]. What does the block output?

Answer
y = [1 + 0.2, 1 − 0.1] = [1.2, 0.9].
A plain layer has slope 0.1. What is the slope of the same layer with a residual connection? Compare the products over 10 such layers.

Answer
1 + 0.1 = 1.1. Over 10 layers: 0.1¹⁰ = 0.0000000001 for the plain stack against 1.1¹⁰ ≈ 2.59 for the residual stack.
Why can't a residual block's branch turn a vector of 64 numbers into one of 128 numbers?

Answer
The block adds x to F(x), entry by entry, and you cannot add vectors of different sizes. The skip path would need its own matrix to change x to 128 numbers first.

Go further

Optional background, for readers who want more than this page.

Dive into Deep Learning: "Residual Networks", sections "Function Classes" and "Residual Blocks" Zhang, Lipton, Li and Smola Textbook section Intermediate

Explains why adding a block that can output the identity never makes a network less capable, with diagrams and code.
Deep Residual Learning for Image Recognition He, Zhang, Ren and Sun (2015) Paper Advanced

The paper that introduced residual networks. Figure 1 shows the deeper plain network training worse; Figure 2 is the residual block.
The Illustrated Transformer: "The Residuals" Jay Alammar Illustrated article Advanced

Shows where the residual connections sit inside a transformer, around attention and the feed-forward network.

Where this shows up

Units that build on this one

The transformer block Advanced