Minibatches: a few examples per step

A small random batch estimates the full gradient

In gradient descent, each step moves a weight w against the gradient g of the loss: w ← w − η × g, where η is the learning rate. The loss is the average of one loss per training example. So its gradient is also an average: the average of the gradients of the single examples.

An average over a million numbers can be estimated from a few of them, picked at random. A minibatch (or just batch) is that random pick. The number of examples in it is the batch size, often between 16 and 1,024. Training with minibatches is called stochastic gradient descent, or SGD; "stochastic" means "random".

Bigger batches are less noisy, but each step costs more

How far a batch gradient lands from the full gradient shrinks as the batch grows. The spread falls with the square root of the batch size: a batch 100 times bigger is 10 times less noisy, and 100 times more work per step. The snippet below measures this on 100,000 made-up examples. It uses a NumPy array and the "spread" of the mean and spread unit.

batch_noise.py

import numpy as np

rng = np.random.default_rng(0)
y = rng.normal(5.0, 2.0, size=100_000)   # one target number per example
w = 0.0                                   # the current weight

full = np.mean(2 * (w - y))               # gradient over all examples
print(f"full gradient: {full:.2f}")

for B in [1, 10, 100, 1000]:
    # y[...] takes the examples at B random positions: one random batch.
    est = [np.mean(2 * (w - y[rng.integers(0, len(y), B)]))
           for _ in range(2000)]          # 2000 random batches of size B
    # {B:4d} pads the whole number B to 4 characters, so the columns line up.
    print(f"batch size {B:4d}: average {np.mean(est):6.2f}, spread {np.std(est):.2f}")

full gradient: -10.00
batch size    1: average -10.05, spread 3.98
batch size   10: average -10.03, spread 1.24
batch size  100: average -10.02, spread 0.39
batch size 1000: average -10.00, spread 0.13

Every batch size gets the average right. Each tenfold increase in batch size cuts the spread by about 3.2, which is √10.

Noisy steps still find the bottom

The figure fits a straight line y = ax + b to 200 points. The rings are lines of equal loss, like the contour lines of a valley on a map. The best (a, b) sits at the center. Each dot is one step of 30, all with the same learning rate, 0.5.

Predict first. With a batch of 1, will the path still get near the center in 30 steps? Then pick each batch size in turn. Find the smallest batch size whose path ends close to the center most times you press "New random batches".

An epoch is one pass through the data

In practice, the program shuffles the training examples into a random order and then cuts that order into consecutive batches. One full pass through all the examples is an epoch. Training usually runs for many epochs, with a new shuffle each time. Each batch processed is one step (also called an iteration).