Cross-entropy: scoring a choice among many

Only the probability given to the right class enters the charge

Picture a model that labels photos as cat, dog or bird. For one photo it says cat 0.659, dog 0.242 and bird 0.099. The photo shows a dog. The model gave the truth a probability of 0.242, so it pays

L = −ln p_right = −ln 0.2424 ≈ 1.417

Had the photo shown a cat, it would pay −ln 0.659 ≈ 0.417. Here ln is the natural logarithm, from the exponentials unit. With two classes this is exactly binary cross-entropy.

Predict first. The photo shows a dog, and the model gives the dog 0.242. Would the charge change if the other 0.758 went all to the bird instead of mostly to the cat?

The full formula compares two lists of probabilities

The textbook formula looks larger than the rule above. Write the truth as a list too: 1 for the right class and 0 for every other class. For a dog this list is (0, 1, 0), which is called a one-hot list. Call it y, and call the model's probabilities p. Then

H(y, p) = −( y₁ ln p₁ + … + y_K ln p_K )

Every term with y_i = 0 vanishes, which leaves −ln p_right. The name comes from information theory, where cross-entropy measures how well one probability list describes another. The full sum matters only when the truth is itself spread over several classes, for example 0.9 on dog and 0.05 on each of the others.

Lowering the charge raises the right score and lowers the others

In practice the loss is computed straight from the scores z that went into the softmax. Taking the logarithm of the softmax formula gives

L = −z_right + ln( e^z₁ + … + e^z_K )

The loss falls when the right class's score rises. It also falls when any other score drops, because that shrinks the sum. For readers who know derivatives: the derivative of L with respect to score i is p_i − y_i. In the dog photo that is +0.659 for cat, −0.758 for dog and +0.099 for bird. Gradient descent moves each score against its derivative. So the dog's score goes up, and each wrong score goes down in proportion to the probability it took.

ce.py

import numpy as np

scores = np.array([[2.0, 1.0, 0.1],    # one row of scores per example
                   [0.5, 2.5, 0.0],
                   [1.0, 1.0, 3.0]])
right = np.array([1, 1, 0])            # the index of the right class

# -log softmax, computed straight from the scores
m = scores.max(axis=1, keepdims=True)
log_p = scores - m - np.log(np.exp(scores - m).sum(axis=1, keepdims=True))
# From each row i, take the entry in column right[i]; np.arange(len(right)) is [0 1 2]
loss = -log_p[np.arange(len(right)), right]
print(np.round(loss, 3))
print(round(loss.mean(), 3))

[1.417 0.197 2.24 ]
1.284

The three rows of scores are the ones behind the worked example's table. The code subtracts each row's largest score first, as the softmax unit explains, so large scores do not overflow.