Gradient descent

The gradient points uphill, so step the other way

A loss function gives one number for how wrong a model is. It depends on the model's weights. Picture the loss as a curve over one weight w. We want the w at the bottom of the curve, but we cannot see the whole curve. We can only compute its slope at the point where we stand.

If the slope is positive, the loss rises to the right, so we move left. If it is negative, we move right. The gradient is that slope; with many weights it is a vector of slopes, one per weight. The update rule is:

w_new = w − η × g

Here g is the gradient at the current w, and η (the Greek letter eta) is the learning rate: a small positive number that scales the step. A steep slope gives a big step; a flat slope gives a small one.

The learning rate decides between crawling and flying off

Try it. The figure runs gradient descent on the same curve, starting at w = 0. Predict first: what happens with a learning rate above 1? Then find a learning rate that gets the loss below 0.01 within five steps.

With many weights, every weight steps at once

A network has many weights. The gradient has one entry for each: how much the loss changes when that weight alone changes a little. One step updates all weights together, each by −η times its own entry. Backpropagation computes all those entries in one pass, and PyTorch does it for you.

In practice, each step computes the loss on a small random batch of examples rather than on all of them. This is called stochastic gradient descent; the unit on minibatches explains it. Improved versions such as AdamW change how each step is sized.

gradient_descent_numpy.py

def loss(w): return (w - 3) ** 2
def grad(w): return 2 * (w - 3)

for lr in [0.1, 1.1]:
    w = 0.0
    print(f"learning rate {lr}")
    for step in range(4):
        w = w - lr * grad(w)
        print(f"  step {step + 1}: w = {w:.4f}, loss = {loss(w):.4f}")

learning rate 0.1
  step 1: w = 0.6000, loss = 5.7600
  step 2: w = 1.0800, loss = 3.6864
  step 3: w = 1.4640, loss = 2.3593
  step 4: w = 1.7712, loss = 1.5099
learning rate 1.1
  step 1: w = 6.6000, loss = 12.9600
  step 2: w = -1.3200, loss = 18.6624
  step 3: w = 8.1840, loss = 26.8739
  step 4: w = -3.2208, loss = 38.6984

The same idea fits a line y = ax + c to four points. PyTorch computes the gradient of both weights; the loop takes the steps. torch.tensor makes a tensor, PyTorch's version of an array. The unit on automatic gradients explains requires_grad, .backward(), torch.no_grad() and .item().

fit_line_torch.py

import torch

x = torch.tensor([0.0, 1.0, 2.0, 3.0])
y = torch.tensor([1.0, 3.0, 5.0, 7.0])     # y = 2x + 1
w = torch.zeros(2, requires_grad=True)     # slope and intercept, both starting at 0

lr = 0.05
for step in range(500):
    loss = ((w[0] * x + w[1] - y) ** 2).mean()
    loss.backward()                  # fills w.grad
    with torch.no_grad():
        w -= lr * w.grad             # one step downhill
        w.grad.zero_()               # clear w.grad, because backward() adds to it
# w.detach(): the numbers in w, without requires_grad=True
print(w.detach(), f"loss {loss.item():.6f}")

tensor([2.0000, 1.0000]) loss 0.000000