Activation functions: bending straight lines
Read these first if they are new to you
- A linear layer: weighted sums Intermediate
The idea in one paragraph
An activation function is a fixed, simple function that a network applies to each output of a linear layer, one number at a time. The most common one, ReLU, replaces negative numbers by zero and keeps positive numbers. Without activation functions, any stack of linear layers works out to a single linear layer, which can only describe straight-line relationships. With them, the stack can bend, and enough bends can follow almost any curve.
Two linear layers in a row are still one straight line
A linear layer computes weighted sums. Take the simplest case: one input x and one output. Let a first layer compute 2x − 1. Let a second layer take that result, multiply by 3 and add 1:
3(2x − 1) + 1 = 6x − 2
The result is again a straight line. This always happens: a weighted sum of weighted sums is a weighted sum. Ten linear layers in a row can do no more than one.
ReLU puts a corner into the line
Now apply a function between the two layers. The ReLU (rectified linear unit) is the simplest:
ReLU(z) = max(0, z)
It keeps a positive number and turns a negative number into 0. So 3 × ReLU(2x − 1) + 1 is flat at 1 until x = 0.5, then rises with slope 6. The line has a corner. The second layer cannot undo that corner, because it only scales and shifts.
A function used this way is called an activation function. It acts on each number of a layer's output separately, and it has no weights of its own to learn.
Take two hidden numbers, h1 = ReLU(x) and h2 = ReLU(x − 1). Combine them with a second linear layer: y = h1 − 2h2.
| x | −1 | 0 | 0.5 | 1 | 2 | 3 |
|---|---|---|---|---|---|---|
| h1 = ReLU(x) | 0 | 0 | 0.5 | 1 | 2 | 3 |
| h2 = ReLU(x − 1) | 0 | 0 | 0 | 0 | 1 | 2 |
| y = h1 − 2h2 | 0 | 0 | 0.5 | 1 | 0 | −1 |
The output is flat, then rises to a peak of 1 at x = 1, then falls. No straight line does that.
Your turn. What is y at x = 1.5?
Answer
h1 = 1.5 and h2 = 0.5, so y = 1.5 − 1 = 0.5. It is on the way down.
Several activation functions are in common use
All of them bend. They differ in shape: sharp corner or smooth curve, and whether the output is limited to a range.
| Name | Formula | Shape |
|---|---|---|
| ReLU | max(0, z) | zero for negatives, a corner at 0 |
| GELU | z × Φ(z) | a smooth version of ReLU that dips slightly below 0 |
| SiLU | z × sigmoid(z) | close to GELU |
| sigmoid | 1 / (1 + e−z) | an S between 0 and 1 |
| tanh | (ez − e−z) / (ez + e−z) | an S between −1 and 1 |
Φ(z) is the chance that a draw from the standard bell curve is below z. The sigmoid has its own unit. ReLU, GELU and SiLU are the usual choices inside networks today. Sigmoid and tanh appear mostly where an output must stay in a range.
Try it. The figure below runs one small network with each activation function in turn. Predict first: which choice gives a straight output line? Then find the choice whose output has sharp corners, and say where they are.
The activation function f(z)
The network's output y(x)
Answer
"None" gives a straight line, y = x − 1.5. ReLU gives corners at x = −1.5, 0 and 1.5, where each hidden number switches on. GELU, sigmoid and tanh give smooth bends near the same places.
Which activation function trains best is mostly found by experiment, not derived. The figure computes GELU with the common tanh approximation, which differs from the exact formula by less than 0.001. ReLU has no slope at exactly 0; software simply uses 0 there.
In code, an activation is one function call
activations.py
import torch
z = torch.tensor([-2.0, -0.5, 0.0, 0.5, 2.0])
print("relu ", torch.relu(z))
print("sigmoid", torch.sigmoid(z))
print("tanh ", torch.tanh(z))
print("gelu ", torch.nn.functional.gelu(z))
relu tensor([0.0000, 0.0000, 0.0000, 0.5000, 2.0000])
sigmoid tensor([0.1192, 0.3775, 0.5000, 0.6225, 0.8808])
tanh tensor([-0.9640, -0.4621, 0.0000, 0.4621, 0.9640])
gelu tensor([-0.0455, -0.1543, 0.0000, 0.3457, 1.9545])The tent from the worked example, with NumPy, next to the same layers without ReLU:
bend.py
import numpy as np
def relu(z):
return np.maximum(0.0, z) # np.maximum keeps the larger of 0 and each number
x = np.array([-1.0, 0.0, 0.5, 1.0, 2.0, 3.0])
# Two linear steps with nothing between: still a straight line.
print(3 * (2 * x - 1) + 1) # same as 6x - 2
# Two hidden units with ReLU, then a weighted sum: a bend.
h1, h2 = relu(x), relu(x - 1)
print(h1 - 2 * h2)
[-8. -2. 1. 4. 10. 16.]
[ 0. 0. 0.5 1. 0. -1. ]Check yourself
-
What is ReLU applied to the list (−3, 0.2, 5)?
Answer
(0, 0.2, 5). The negative number becomes 0; the others stay.
-
A network has five linear layers and no activation functions. Can it fit a set of points that lie on a parabola?
Answer
No. The five layers work out to one linear layer, so the output is a straight line (a flat plane, with more inputs). It cannot follow the curve.
-
Using the tent network y = ReLU(x) − 2 × ReLU(x − 1), find y at x = 4 and at x = −5.
Answer
At 4: 4 − 2 × 3 = −2. At −5: both hidden numbers are 0, so y = 0.
-
A model's output must be a probability between 0 and 1. Which activation from the table would you put on that output, and why not ReLU?
Answer
The sigmoid, because its output always lies between 0 and 1. ReLU can give any number of 0 or above, including numbers above 1.
Go further
Optional background, for readers who want more than this page.
-
A short tour of sigmoid, tanh, ReLU and its variants, with their known problems in training.
-
Plots each function next to its slope, with runnable code.
-
Switch the activation menu between "Linear" and "ReLU" and watch which shapes a network can then separate.
Where this shows up
Units that build on this one
- The multilayer perceptron (MLP) Intermediate