Concept units

The multilayer perceptron (MLP)

Intermediate

Read these first if they are new to you

The idea in one paragraph

A multilayer perceptron, or MLP, is a stack of linear layers with an activation function after each one except the last. Numbers flow in at one end and come out at the other. The middle layers are called hidden layers. Each activation adds bends, so a wide enough MLP can follow almost any smooth relationship between its inputs and its outputs. MLPs are the basic building block of neural networks, and they appear inside most larger designs.

An MLP alternates linear layers and activation functions

A linear layer computes weighted sums. An activation function, such as ReLU, bends each number afterwards. An MLP simply repeats the pair:

input → linear → ReLU → linear → ReLU → linear → output

The lists of numbers between the input and the output are the hidden layers. They are called hidden because the training data gives no values for them: it fixes only the inputs and the targets. The length of a hidden list is the layer's width. The number of linear layers is the network's depth.

An MLP with 2 inputs, two hidden layers of width 4, and 1 output
Every number in one column feeds every number in the next. Each gap between columns is a linear layer. The first two gaps are followed by ReLU; the last is not. This network has 37 parameters: 2 × 4 + 4, then 4 × 4 + 4, then 4 × 1 + 1.
Worked example: one pass through a tiny MLP

The network has 2 inputs, one hidden layer of width 2 with ReLU, and 1 output. Its parameters are:

  • first layer: W1 = [[1, −1], [0.5, 1]], b1 = (0, −1);
  • second layer: W2 = [[2, 1]], b2 = 0.5.

Feed in x = (1, 2).

  1. First layer: z1 = 1 × 1 − 1 × 2 + 0 = −1 and z2 = 0.5 × 1 + 1 × 2 − 1 = 1.5.
  2. ReLU: h = (max(0, −1), max(0, 1.5)) = (0, 1.5). The first hidden number is switched off.
  3. Second layer: y = 2 × 0 + 1 × 1.5 + 0.5 = 2.

Your turn. Feed in x = (2, 0). What is y?

Answer

z = (2, 0), so h = (2, 0) and y = 2 × 2 + 0 + 0.5 = 4.5.

More hidden units give more bends

Each hidden unit with ReLU switches on at one input value, which adds one corner to the output. With enough corners, the output can trace a curve as closely as you like. The figure below shows this with weights chosen by hand. Training finds its own weights, but the pieces it builds with are the same.

Try it. The MLP has one input, one hidden layer with ReLU, and one output. It tries to follow the curve sin(x). Predict how many hidden units it needs before the largest gap between the two curves is below 0.05. Then move the slider to find the fewest that do it.

A one-hidden-layer MLP following sin(x)
largest gap -
target sin(x)MLP outputwhere a hidden unit switches on
The output is made of straight pieces, one more for each hidden unit. Hidden unit i computes ReLU(x − ti), so it switches on at ti. The output layer weights each one by how much the slope must change there.
Answer

10 hidden units. The largest gap is about 0.049 with 10 and about 0.059 with 9. Halving the gap takes about 1.4 times as many units here, because the gap shrinks with the square of the piece length.

"Almost any relationship" has limits

The universal approximation theorem says that one wide enough hidden layer can approximate any continuous function on a bounded range. It says such weights exist. It does not say how many units are needed, or that training will find them. In practice, deeper networks often need far fewer units than one very wide layer.

The last layer usually has no activation

The output layer is left linear so that the output can take any value: negative, positive, large or small. A ReLU there would forbid negative outputs. When an output must be a probability, a later step such as the sigmoid or softmax turns it into one.

In PyTorch, an MLP is a list of layers run in order:

mlp_torch.py

import torch
from torch import nn   # so that nn.Linear means torch.nn.Linear

mlp = nn.Sequential(
    nn.Linear(2, 4), nn.ReLU(),
    nn.Linear(4, 4), nn.ReLU(),
    nn.Linear(4, 1),
)
# p.numel() counts the numbers in p, so the sum is the number of parameters
print(sum([p.numel() for p in mlp.parameters()]))

points = torch.randn(5, 2)   # a batch of 5 random inputs, 2 numbers each
print(mlp(points).shape)

37
torch.Size([5, 1])

The worked example, step by step in NumPy:

mlp_numpy.py

import numpy as np

x  = np.array([1.0, 2.0])
W1 = np.array([[1.0, -1.0],
               [0.5,  1.0]])
b1 = np.array([0.0, -1.0])
W2 = np.array([[2.0, 1.0]])
b2 = np.array([0.5])

z = W1 @ x + b1          # first linear layer
h = np.maximum(0.0, z)   # ReLU
y = W2 @ h + b2          # second linear layer
print(z, h, y)

[-1.   1.5] [0.  1.5] [2.]

Check yourself

  1. An MLP takes 3 inputs, has one hidden layer of width 8, and gives 1 output. How many parameters does it have?

    Answer

    First layer 3 × 8 + 8 = 32. Second layer 8 × 1 + 1 = 9. Total 41.

  2. Using the worked example's network, what is y for x = (0, 0)?

    Answer

    z = b1 = (0, −1), so h = (0, 0) and y = b2 = 0.5.

  3. An MLP predicts tomorrow's temperature in degrees Celsius. Someone puts a ReLU after its last layer. What goes wrong?

    Answer

    The output can never be below 0, so the network cannot predict a frost. The last layer should stay linear.

  4. Someone deletes every ReLU from an MLP with four linear layers. What can the result still do?

    Answer

    Only what one linear layer can do: the four layers work out to a single weighted sum plus a constant, so every output is a straight-line (flat) function of the inputs.

Go further

Optional background, for readers who want more than this page.

Where this shows up

Units that build on this one