Softmax: probabilities over many choices

Softmax turns scores into shares of a whole

Picture a model that looks at a photo and must say whether it shows a cat, a dog or a bird. It gives each choice, or class, a score: say 2.0 for cat, 1.0 for dog and 0.1 for bird. These scores rank the classes, but they are not probabilities. A score can be negative, and the three do not add up to 1.

Softmax fixes both problems in two steps. First, raise e ≈ 2.718 to each score. The exponentials unit shows that e^z is positive for every z. Second, divide each result by the sum of all of them. For class i out of n classes:

p_i = e^z_i / ( e^z₁ + e^z₂ + … + e^z_n )

Predict first. The cat's score is 1 higher than the dog's. Will the cat be twice as likely as the dog, or more, or less?

Only the gaps between scores matter

Divide the formula for one class by the formula for another. The totals cancel, and what is left is

p_i / p_j = e^{z_i − z_j}

So each extra point of score gap multiplies the ratio of the two probabilities by e ≈ 2.718. Three things follow:

• Adding the same number to every score leaves every probability unchanged, because no gap changes.
• Doubling every score doubles every gap, so the largest class takes a bigger share. Scores 4.0, 2.0 and 0.2 give 0.864, 0.117 and 0.019.
• Halving every score shrinks the gaps, so the shares even out. Scores 1.0, 0.5 and 0.05 give 0.502, 0.304 and 0.194.

When the gaps are large, almost all the probability goes to the top score. That is why the function is called a "soft" maximum: it behaves like picking the largest score, but smoothly. Some code divides every score by a number called the temperature before the softmax, to make the shares sharper or flatter on purpose.

With exactly two classes, softmax is the sigmoid in disguise. Give one class score z and the other 0. Then the first class gets e^z / (e^z + 1) = 1 / (1 + e^−z) = σ(z).

softmax.py

import numpy as np

def softmax(z):
    e = np.exp(z - z.max())   # subtract the largest score first
    return e / e.sum()

print(np.round(softmax(np.array([2.0, 1.0, 0.1])), 3))
print(np.round(softmax(np.array([12.0, 11.0, 10.1])), 3))
print(np.round(softmax(np.array([1000.0, 999.0])), 3))

[0.659 0.242 0.099]
[0.659 0.242 0.099]
[0.731 0.269]

Softmax also appears inside networks, where it turns similarity scores into weights. The attention unit covers that use. To train a model whose output is a softmax, the usual loss is cross-entropy.