Loss: one number for how wrong

A loss turns many errors into one number

A model makes a prediction for each example. Each example also has a target: the right answer. The error is prediction minus target. A model makes many predictions, so it has many errors. Some are positive and some negative.

To say whether one setting of the weights is better than another, we need a single number. A loss function provides it. It must be 0 when every prediction is right, and grow as the predictions get worse.

Mean squared error averages the squared gaps

The mean squared error (MSE) squares each error and takes the mean. For n examples with predictions p_i and targets t_i:

MSE = (1/n) ∑ (p_i − t_i)²

Squaring makes every term positive, so errors above and below cannot cancel. A related loss, the mean absolute error (MAE), averages |p_i − t_i| instead.

Try it. The figure fits a straight line to five points, and each squared error is drawn as a square whose side is the gap. The MSE is the average area of the squares. Predict first: will the best line pass through every point? Then move the slider until the MSE is as small as you can make it.

Squaring punishes one big error more than many small ones

Compare two models on ten examples. Model A is off by 1 on every example. Model B is exactly right on nine and off by 10 on one.

MAE rates the two models the same. MSE rates B ten times worse. So training with MSE works hard to avoid any large error, and a few wrong targets in the data can pull the model a long way. MAE is calmer about such outliers.

When the answer is a point, the error is a distance

Many models predict several numbers at once, such as the coordinates of a point. Then the natural error is the distance between the predicted point and the target point. The squared distance is the sum of the squared coordinate errors. Averaged over examples, it gives an MSE for points.

When one input has several targets, the mean is the best single prediction

Sometimes the same input comes with different targets. The model can give only one prediction for that input, so it must pick one number c for all of those targets. Take the five targets 4, 8, 6, 5 and 7, the film ratings from summarizing a list of numbers. Which c makes the MSE smallest?

The answer is their mean, 6. The MSE splits into two parts:

MSE(c) = (c − mean)² + variance

To see why, write each error as (c − mean) + (mean − target) and square it. The cross terms cancel, because the deviations from the mean add up to zero. The variance does not depend on c. So the MSE is smallest when the first part is 0, that is, when c is the mean.

Two facts follow. First, when the targets disagree, the loss cannot reach 0: the best it can do is their variance. Second, the best prediction need not be any of the targets: for the targets −1 and +1 it is 0. For points, the same holds for each coordinate, so the best single point is the average point. MAE behaves differently: its best single prediction is the median, the middle value.

The loss is what training sees

Training only makes the loss small. It does not see anything else. So the loss must change smoothly as the weights change, or gradient descent cannot tell which way to move them. This is why classifiers train on cross-entropy rather than on the share of correct answers, which changes in jumps.

A small loss on the training examples is also no promise about new examples. A validation set checks that.

loss_numpy.py

import numpy as np

target = np.array([3.0, 5.0, 7.0])
pred   = np.array([2.5, 5.0, 8.0])

err = pred - target
print(err)
print("MSE", np.mean(err ** 2))
print("MAE", np.mean(np.abs(err)))

[-0.5  0.   1. ]
MSE 0.4166666666666667
MAE 0.5

PyTorch has both losses built in. MAE is called L1Loss there.

loss_torch.py

import torch

target = torch.tensor([3.0, 5.0, 7.0])   # a tensor: PyTorch's kind of array
pred   = torch.tensor([2.5, 5.0, 8.0])
# MSELoss() makes the loss function; (pred, target) then calls it
print(torch.nn.MSELoss()(pred, target))
print(torch.nn.L1Loss()(pred, target))

tensor(0.4167)
tensor(0.5000)