Binary cross-entropy: scoring yes-or-no guesses
Read these first if they are new to you
- The sigmoid: from score to probability Intermediate
- Loss: one number for how wrong Intermediate
The idea in one paragraph
A model that answers yes or no gives a probability of yes, usually from a sigmoid. Binary cross-entropy is the loss that grades that probability once the true answer is known. The charge is minus the natural logarithm of the probability the model gave to what really happened. A confident right answer costs almost nothing. A confident wrong answer costs a lot, and the charge has no upper limit.
The charge is minus the log of the probability given to what happened
Picture a weather forecaster who says "90% chance of rain". The next day it either rains or it does not. How should we score the forecast? A good rule rewards forecasts that put high probability on what happened. It should punish a forecaster who was sure and wrong more than one who was unsure.
Binary cross-entropy does this. Call the true answer y: 1 for yes, 0 for no. Call the model's probability of yes p. The charge for one example is
L = −[ y ln p + (1 − y) ln(1 − p) ]
Only one of the two terms is ever switched on. If the answer is yes (y = 1), the charge is −ln p. If the answer is no (y = 0), the charge is −ln(1 − p). Either way, it is minus the log of the probability the model gave to the true answer. Here ln is the natural logarithm, from the exponentials unit.
Predict first. The forecaster said 90% rain. Guess which charge is larger, and by how much: the one for "it rained" or the one for "it stayed dry"?
The forecaster said p = 0.9.
- It rained (y = 1): charge = −ln 0.9 ≈ 0.105.
- It stayed dry (y = 0): charge = −ln 0.1 ≈ 2.303, about 22 times more.
- A forecaster who said 0.5 pays −ln 0.5 ≈ 0.693 either way.
To grade a model on many examples, average the charges. Four examples:
| True answer y | Model's p of yes | Probability of what happened | Charge |
|---|---|---|---|
| 1 (yes) | 0.9 | 0.9 | 0.105 |
| 0 (no) | 0.2 | 0.8 | 0.223 |
| 1 (yes) | 0.6 | 0.6 | 0.511 |
| 0 (no) | 0.95 | 0.05 | 2.996 |
The average is (0.105 + 0.223 + 0.511 + 2.996) / 4 ≈ 0.959. The one confident mistake in the last row supplies more than three quarters of the total.
Try it. With the answer set to no, drag p from 0.5 up to 0.9. Then guess the charge at 0.99 before you drag there. Does moving from 0.9 to 0.99 add more or less than moving from 0.5 to 0.9?
The logarithm makes the loss add up across examples
Why a logarithm and not something simpler? Treat each answer as independent. Then the probability the model gives to the whole set of true answers is the product of its probabilities for each one. A good model makes that product large. A logarithm turns a product into a sum, and a sum is easy to average and to improve one example at a time. So making the average binary cross-entropy small is the same as making the true answers as likely as possible under the model. Statisticians call this maximum likelihood.
The logarithm also explains the missing ceiling. A squared-error loss, (y − p)2, charges at most 1 for any mistake. Binary cross-entropy charges 4.6 for a probability of 0.01 on the truth, and 9.2 for 0.0001. A model trained with it learns that confident mistakes are very expensive.
bce.py
import numpy as np
y = np.array([1, 0, 1, 0]) # the right answers: 1 = yes, 0 = no
p = np.array([0.9, 0.2, 0.6, 0.95]) # the model's probability of yes
loss = -(y * np.log(p) + (1 - y) * np.log(1 - p))
print(np.round(loss, 3))
print(round(loss.mean(), 3))
[0.105 0.223 0.511 2.996] 0.959
If a model gives p = 0 exactly and the answer is yes, ln 0 is minus infinity and the loss breaks.
With a sigmoid, p never reaches 0 or 1 in exact arithmetic, but a computer can round it there.
So libraries start from the model's score z, before the sigmoid. For a true yes,
−ln σ(z) equals ln(1 + e−z). Written that way, it still overflows
when z is very negative. So libraries compute it as
max(−z, 0) + ln(1 + e−|z|), whose exponent is never positive.
PyTorch calls this version binary_cross_entropy_with_logits. The value is the same; only
the arithmetic is safer.
Binary cross-entropy is the two-choice case of cross-entropy, which grades a choice among many.
Check yourself
-
A model gives p = 0.99 that an email is spam. It is not spam. What is the charge?
Answer
The model gave 1 − 0.99 = 0.01 to the true answer, so the charge is −ln 0.01 ≈ 4.605. Compare 0.010 if the email had been spam.
-
A lazy model says p = 0.5 for every example. What is its average binary cross-entropy? Why is that number a useful yardstick?
Answer
Every example costs −ln 0.5 = ln 2 ≈ 0.693, so the average is 0.693. A model whose average is not below 0.693 has learned nothing useful about the answers. This yardstick assumes yes and no are equally common. If 90% of the answers are yes, always saying 0.9 already scores about 0.325, so a useful model must beat that instead.
-
Which costs more in total? (a) Two yes-examples, each given p = 0.6. (b) Two yes-examples, one given 0.99 and one given 0.2.
Answer
(a) costs 2 × 0.511 = 1.022. (b) costs 0.010 + 1.609 = 1.619. One confident miss outweighs one confident hit, so (b) costs more.
-
Can a model with a sigmoid output ever reach a loss of exactly 0 on a yes-example?
Answer
No. That needs p = 1, and the sigmoid only approaches 1 as the score grows. The loss keeps shrinking toward 0, so training keeps nudging scores of easy examples higher, by ever smaller amounts.
Go further
Optional background, for readers who want more than this page.
-
Logistic regression: loss and regularization Google Machine Learning Crash Course Course page Intermediate
The "Log Loss" section gives the same formula under its other common name and says why squared error is a poor fit here.
-
Its "Estimating Coefficients" section shows how this loss chooses a model's weights.
Where this shows up
No unit or explanation builds on this one yet.