Held-out data: the validation set
Read these first if they are new to you
- Loss: one number for how wrong Intermediate
- Gradient descent Intermediate
The idea in one paragraph
A model can lower its loss on the training examples by memorizing them, noise and all. Memorizing does not help on new examples. To tell learning from memorizing, set aside some examples that the model never trains on: the validation set. Check the loss on them now and then. While both losses fall, the model is learning. When the training loss keeps falling but the validation loss rises, the model has started to memorize.
Split the data before training starts
Divide the examples at random into three parts, before any training:
- The training set, often about 80% of the examples. Training steps use only these.
- The validation set, often about 10%. You measure the loss on it during training, to decide when to stop and which settings work best. The model never takes a step on it.
- The test set, the rest. You measure on it once, at the very end, to report how well the finished model does.
The test set exists because choosing by the validation loss many times slowly tunes your choices to those particular validation examples. A set that took no part in any choice gives an honest final score.
split.py
import numpy as np
rng = np.random.default_rng(0)
n = 10_000 # examples, numbered 0 to 9999
order = rng.permutation(n) # shuffle first, so each part is a fair sample
train, val, test = order[:8000], order[8000:9000], order[9000:]
print(len(train), len(val), len(test))
# np.intersect1d(a, b) holds the numbers that are in both a and b
print("shared examples:", len(np.intersect1d(train, val)), len(np.intersect1d(val, test)))
print("first five validation examples:", val[:5])
8000 1000 1000
shared examples: 0 0
first five validation examples: [3545 8124 8713 31 9368]Shuffle before you split. If the examples were stored in some order, such as by date or by source, a split without shuffling would put different kinds of examples in each part.
Rising validation loss means the model is memorizing
The figure trains a flexible curve on 12 points. The curve has 12 adjustable numbers, enough to pass through all 12 points exactly. The points are a smooth wave plus random noise. Another 40 points from the same wave, with their own noise, form the validation set. Training is plain gradient descent on the training points only.
Fitting the noise of the training examples, so that the model does worse on new ones, is called overfitting. A model that does well on new examples generalizes.
Predict first. As training goes on, does the validation loss keep falling? Then drag the slider until the validation loss is at its lowest. Keep going, and watch what the curve does between the training points.
These are the losses from the figure at a few of the epochs the slider shows.
| Epoch | Training loss | Validation loss |
|---|---|---|
| 11 | 0.391 | 0.338 |
| 81 | 0.032 | 0.104 |
| 930 | 0.007 | 0.144 |
| 10,685 | 0.004 | 0.265 |
| 200,000 | 0.003 | 0.769 |
- The training loss falls at every row. By itself, it says that more training is always better.
- Of the epochs the slider shows, the validation loss is lowest at epoch 81, at 0.104. After that it rises, to more than 7 times as much by the last row.
- So keep the model from epoch 81. Stopping at the lowest validation loss, or keeping a saved copy from that point, is called early stopping.
Your turn. A run logs validation losses 0.50, 0.31, 0.27, 0.29, 0.34 at epochs 1 to 5. Which epoch's model do you keep? (Answer: epoch 3, at 0.27.)
This toy has 12 training points, so it overfits fast and clearly. Real validation curves are noisier, and they can flatten rather than rise. Some measures, such as the share of correct answers, can keep improving while the validation loss creeps up. Very large models sometimes improve again after a bump. So treat the validation loss as evidence to read, not as a rule that decides alone.
The validation set must be truly unseen
The validation loss is honest only if no validation example, or a near copy of one, is in the training set. Near copies are common: two crops of the same picture, or two pieces of the same object. Split by the larger unit (the picture, the object) so that all its pieces land on the same side. When information from the validation set reaches training, it is called a leak, and the validation loss looks better than the model really is.
Check yourself
-
A model's training loss is 0.02 and its validation loss is 0.95. A second model's are 0.20 and 0.25. Which would you rather use on new data, and why?
Answer
The second. Its validation loss, 0.25, is much lower, and validation loss is the estimate for new data. The first model's large gap between 0.02 and 0.95 is the sign of overfitting.
-
You try 20 learning rates and keep the one with the lowest validation loss. Why should you still report the test loss, not that validation loss?
Answer
You picked the winner because it did best on the validation set, partly by luck. That makes its validation loss slightly too good. The test set took no part in the choice, so its loss is an honest estimate.
-
Your data holds 50 photos of each of 200 birds. You split the 10,000 photos at random. Why might the validation loss look too good?
Answer
Photos of the same bird land in both sets, so the model can recognize a bird it trained on. Split by bird instead: all 50 photos of a bird go to the same set.
Go further
Optional background, for readers who want more depth.
-
A scrolling explainer with one section per set: "The Training Set", "The Validation Set", "The Testing Set".
-
Reads training and validation loss curves like the one above, with exercises.
Where this shows up
Units that build on this one
- Overfitting: when a model fits the noise Intermediate
- The training loop Intermediate