Concept units

The gradient: the uphill direction

Intermediate

Read these first if they are new to you

The idea in one paragraph

A function with several inputs is like a landscape: the inputs say where you stand and the output is the height. At any spot you can ask for the slope in each input's direction, holding the other inputs still. Put those slopes together in a vector and you have the gradient. It points in the direction that climbs fastest, and its length says how steep that climb is. Its opposite points downhill fastest, which is how many programs search for the lowest point.

One slope per input, holding the others still

Picture a valley. Your position is two numbers, x (east) and y (north), and the height of the ground is f(x, y) = x2 + 3y2. The bottom is at (0, 0), and the valley is steeper going north than going east.

Stand somewhere and take a small step due east. Only x changes, so the height behaves like a one-input curve, and its slope is an ordinary derivative. That slope is the partial derivative with respect to x, written ∂f/∂x (the curly ∂ says "the other inputs are held still"). Do the same due north to get ∂f/∂y.

For this valley, holding y still makes 3y2 a constant, so ∂f/∂x = 2x. Holding x still gives ∂f/∂y = 6y.

The gradient is the vector of these slopes, one per input. It is written ∇f (say "grad f"):

∇f(x, y) = (∂f/∂x, ∂f/∂y) = (2x, 6y)

Worked example

Find the gradient of f(x, y) = x2 + 3y2 at the point (1, 1).

  1. East slope: 2x = 2 × 1 = 2.
  2. North slope: 6y = 6 × 1 = 6.
  3. So ∇f(1, 1) = (2, 6). Uphill is mostly north, because the valley wall is steeper that way.
  4. Check the east slope with a small step: f(1.01, 1) = 1.0201 + 3 = 4.0201, and f(1, 1) = 4. The change 0.0201 divided by 0.01 is 2.01, close to 2.

Your turn. Find the gradient of the same function at (−2, 0.5). Which way is downhill?

Answer

(2 × (−2), 6 × 0.5) = (−4, 3). Downhill fastest is the opposite arrow, (4, −3): east and south, more east than south. It does not point straight at the bottom, (0, 0). This valley is steeper from north to south than from east to west, so the steepest way down leans more to the south than the straight line to the bottom does.

The gradient points uphill and crosses the height lines at right angles

A map can show a landscape with contour lines: each line joins points of the same height. Walking along a contour line, you neither climb nor descend. The steepest way up is straight across the lines, and that is the direction the gradient points.

The gradient arrow points straight up the slope, across the contour lines
Point (1.0, 1.0), height 4.00
gradient = (2.0, 6.0), steepness 6.32
Try this. Each oval joins points of equal height; the outer ones are higher. The orange arrow is the gradient, drawn at a fifth of its length. The dashed blue arrow points the opposite way, downhill. Drag the point. Goal: find the one spot where the gradient vanishes, and check that the arrow always crosses the ovals at a right angle.

The length of the gradient tells you how steep the steepest climb is. Near the bottom of the valley, the ground is nearly flat and the gradient is short. At the very bottom it is (0, 0): there is no uphill direction. A program that wants the lowest point can take small steps against the gradient, one after another. That method is gradient descent.

This is a simplification

The gradient describes the ground right under your feet. "Uphill fastest" is exact only for a very small step. A long step along the gradient can overshoot and land higher or lower than you meant. A gradient of zero also does not prove you are at the bottom: the top of a hill and a saddle between two hills are flat too.

grad.py: estimate each partial derivative with a small step on each side

import numpy as np

def f(p):
    x, y = p
    return x ** 2 + 3 * y ** 2

def numerical_gradient(f, p, h=1e-6):   # 1e-6 means 0.000001
    g = np.zeros(len(p))               # an array of zeros, one per input
    for i in range(len(p)):
        step = np.zeros(len(p))
        step[i] = h
        g[i] = (f(p + step) - f(p - step)) / (2 * h)
    return g

p = np.array([1.0, 1.0])
print(np.round(numerical_gradient(f, p), 4))   # exact answer: [2, 6]

[2. 6.]

This takes two function evaluations per input. A function with a million inputs would need two million, which is why real programs compute gradients exactly with the chain rule instead.

Check yourself

  1. For f(x, y) = x2 + y2, what is the gradient at (3, 4)? Which direction is downhill fastest?

    Answer

    (6, 8). Downhill fastest is (−6, −8), straight back toward the origin.

  2. What is the gradient of f(x, y) = 2x + 5y? Why does it not depend on the point?

    Answer

    (2, 5) everywhere. The surface is a flat tilted plane, so its slope is the same at every spot.

  3. You walk along a contour line. What angle does your direction make with the gradient?

    Answer

    A right angle. You neither climb nor descend, so your step has no part along the uphill direction.

Go further

Optional background, for readers who want more than this page.

Where this shows up

Units that build on this one