Overfitting: when a model fits the noise

A flexible model can fit the noise along with the pattern

Take 12 training points, evenly spaced from x = −1 to x = 1. Each y is a smooth sine wave, sin(2.5x), plus random noise with a standard deviation of 0.25. Another 40 points from the same wave, at random x and with their own noise, form the validation set.

The model is a curve with 12 weights:

y = w₀ + w₁x + w₂x² + … + w₁₁x¹¹

With 12 weights and 12 points at different x, one choice of weights makes the curve pass through every training point exactly. Its training loss, the mean squared error (MSE) on those points, is 0. But its validation loss is 6.36. The wave itself scores 0.070 on the validation points, close to 0.0625, which is 0.25²: the noise alone.

Fitting the noise of the training examples, so that the model does worse on new ones, is called overfitting. The opposite failure is underfitting: a model too stiff to follow the pattern, so that both losses stay high. A sign of overfitting is a low training loss with a validation loss that is much higher, and higher than a calmer model gives.

The exact fit uses huge weights that nearly cancel: w₉ is about −2,333 and w₇ about 1,822. At the training points, the terms cancel to give each y. Between the points they do not, and the curve swings far from the wave.

A penalty on the squared weights makes large weights cost loss

To regularize the fit, train on a total that adds a penalty to the training MSE:

total = MSE + (λ/2) × (w₁² + w₂² + … + w₁₁²)

The number λ (lambda) is the regularization strength, zero or more. You choose it; training does not. At λ = 0 there is no penalty. The larger λ is, the more loss each large weight costs, so training accepts a less exact fit to keep the weights small. This penalty is called L2 regularization, or ridge regression in statistics. The ½ only makes the step formula below tidier.

The penalty leaves out w₀. That weight moves the whole curve up or down, which cannot make it swing. A penalty on it would only pull the curve toward y = 0. Leaving out such a constant term, called a bias in a network, is the common practice.

Predict first. The slider starts at λ = 0, the exact fit. As you raise λ, will the training loss rise or fall? Then find the λ with the lowest validation loss, and keep going to see what a very strong penalty does.

This code computes the figure's fits for a few strengths. Setting every slope of the total to zero gives 12 linear equations in the 12 weights, and np.linalg.solve solves them. So it finds the lowest total directly, without gradient-descent steps. In the code, X_tr @ w multiplies a matrix by a vector, as in Multiplying a matrix by a vector. X_tr.T is X_tr with its rows and columns swapped, and @ also multiplies two matrices.

ridge_fit.py

import numpy as np
rng = np.random.default_rng(4)
x_tr = np.linspace(-1, 1, 12)                  # 12 training inputs, evenly spaced
y_tr = np.sin(2.5 * x_tr) + 0.25 * rng.normal(size=12)
x_va = rng.uniform(-1, 1, 40)                  # 40 random validation inputs from -1 to 1
y_va = np.sin(2.5 * x_va) + 0.25 * rng.normal(size=40)

X_tr = np.vander(x_tr, 12, increasing=True)    # row i holds 1, x, x**2, ..., x**11
X_va = np.vander(x_va, 12, increasing=True)
P = np.eye(12)                                 # 12 × 12: 1 on the diagonal, 0 elsewhere
P[0, 0] = 0                                    # the penalty leaves out w0

print("lambda    train    valid  sum of squares")
for lam in [0, 1e-6, 1e-4, 1e-3, 1e-2, 1e-1, 1, 10]:
    # The weights with the lowest total: training MSE + (lam / 2) * (w1**2 + ... + w11**2).
    w = np.linalg.solve(X_tr.T @ X_tr + 6 * lam * P, X_tr.T @ y_tr)
    train = np.mean((X_tr @ w - y_tr) ** 2)
    valid = np.mean((X_va @ w - y_va) ** 2)
    # The g in :6g and :14.4g picks plain or e notation by the value's size.
    # .4g rounds to 4 significant digits.
    print(f"{lam:6g}  {train:7.4f}  {valid:7.4f}  {np.sum(w[1:] ** 2):14.4g}")

lambda    train    valid  sum of squares
     0   0.0000   6.3556       1.013e+07
 1e-06   0.0102   0.1214            3248
0.0001   0.0221   0.0845           86.14
 0.001   0.0321   0.0802            10.5
  0.01   0.0424   0.0895           3.076
   0.1   0.0748   0.1336           1.146
     1   0.2218   0.3023           0.203
    10   0.4727   0.5454         0.01138

The last column is w₁² + … + w₁₁², and 1.013e+07 means 1.013 × 10⁷. The 6 in the code is 12 points divided by 2, from setting the slopes to zero.

With gradient descent, the penalty shrinks each weight a little at every step

Look at the penalty's part for one weight w: (λ/2) × w². The slope of w² is 2w, as the unit on derivatives shows, and multiplying a curve by a number multiplies its slope by the same number. So the penalty's slope is λw. If g is the gradient of the MSE for this weight, the gradient of the total is g + λw. One gradient-descent step with learning rate η is then:

w_new = w − η × (g + λw) = w × (1 − ηλ) − η × g

The right-hand form says: shrink the weight by the factor 1 − ηλ, then take the usual step. That shrinking is the weight decay of the AdamW unit, with the same formula. When ηλ is below 1, as it is in practice, the factor pulls a weight toward zero from either side without making it exactly zero.

A penalty is one of several ways to curb overfitting

Each of these often lowers the validation loss too:

• More training examples. A curve with 12 weights cannot pass through all of 100 noisy points.
• Fewer weights. A curve with 5 weights turns at most 3 times, against 10 for 12 weights. Too few weights underfit.
• Early stopping. Stop where the validation loss is lowest, as the validation-set unit shows.
• Data augmentation. Train on altered copies of each example, such as rotated ones.

By the broad definition in the idea paragraph, early stopping and data augmentation count as regularization too.