Concept units

The sigmoid: from score to probability

Intermediate

Read these first if they are new to you

The idea in one paragraph

Many models answer a yes-or-no question with a score: any number, negative or positive, where higher means "more likely yes". The sigmoid turns that score into a probability between 0 and 1. A score of 0 becomes 0.5. Large positive scores come close to 1, and large negative scores come close to 0. Keep the S-shaped curve below in mind: flat at both ends, steep in the middle.

A score can be any number, but a probability must lie between 0 and 1

Picture a spam filter. It reads an email and adds up evidence: a strange link adds to the score, a known sender subtracts from it. The result might be 2, or −7, or 0.3. That number ranks emails well, but you cannot read it as a chance. "Score 2" does not tell you how sure the filter is.

The sigmoid fixes this. It takes the score z and returns

σ(z) = 1 / (1 + e−z)

Here e ≈ 2.718 is the base of natural exponentials, from the exponentials unit. The Greek letter σ (sigma) is the usual name for the function. Libraries often call the score a logit; this page says "score".

Why does this formula stay between 0 and 1? The term e−z is always positive. So the bottom, 1 + e−z, is always bigger than 1, and one divided by it is always between 0 and 1.

Predict first. Before you read the example, guess the probability for a score of 0.

Worked example: a score of 2
  1. Flip the sign of the score: −2.
  2. Raise e to that power: e−2 ≈ 0.1353.
  3. Add 1: 1.1353.
  4. Divide 1 by it: 1 / 1.1353 ≈ 0.8808.

So a score of 2 means about an 88% chance of yes. For a score of 0, e0 = 1, and σ(0) = 1 / 2 = 0.5, an even chance. For a score of −2 the answer is 0.1192, which is 1 − 0.8808.

The sigmoid squeezes every score into the band from 0 to 1
1 0.5 0 −6 −3 0 3 6 score z probability σ(z)
σ(2.0) = 0.881

Try it. Drag until the probability reaches 0.95. What score did you need? Then find the score for 0.05.

What to notice. Near score 0 a small change in score moves the probability a lot. Beyond about ±4 the curve is nearly flat, so even a big change in score hardly moves the probability. (A probability of 0.95 needs a score of about 2.94; 0.05 needs about −2.94.)

The score is the log of the odds

The sigmoid has an exact inverse. The odds of an event are its probability divided by the probability that it does not happen: p / (1 − p). A probability of 0.8 gives odds of 4, or "4 to 1". The score that the sigmoid turns into p is the natural logarithm of those odds:

z = ln( p / (1 − p) )

Check it on the worked example. For p = 0.8808 the odds are 0.8808 / 0.1192 ≈ 7.39, and ln 7.39 ≈ 2, the score we started from. So adding 1 to a score multiplies the odds by e ≈ 2.718, wherever you start on the curve.

Two more facts follow from the formula. First, the curve is symmetric: σ(−z) = 1 − σ(z). Second, its slope is steepest at z = 0, where it equals 1/4.

sigmoid.py

import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

scores = np.array([-4.0, -2.0, 0.0, 2.0, 4.0])
p = sigmoid(scores)
print(np.round(p, 4))

# Undo the sigmoid: the log-odds of each probability.
print(np.round(np.log(p / (1 - p)), 4))

[0.018  0.1192 0.5    0.8808 0.982 ]
[-4. -2.  0.  2.  4.]
A number between 0 and 1 is not automatically a good probability

The sigmoid only guarantees the range. Whether "0.9" really comes true 9 times in 10 depends on how the model was trained. Training with binary cross-entropy pushes the outputs toward honest probabilities, but a model can still be over-confident on data unlike its training data.

There is also a computing detail. For a very negative score, such as −1000, e1000 is too large for a computer's numbers, and the simple formula above overflows. NumPy prints a warning and returns 0. Libraries use a rearranged formula that avoids this.

The same S-shaped curve also appears inside neural networks, where it bends a weighted sum. The activation functions unit covers that use.

Check yourself

  1. σ(3) ≈ 0.9526. Without a calculator, what is σ(−3)?

    Answer

    1 − 0.9526 = 0.0474. The curve is symmetric about the point (0, 0.5).

  2. A model should say "75% chance of yes". What score must it produce?

    Answer

    The odds are 0.75 / 0.25 = 3, so the score is ln 3 ≈ 1.10. Check: 1 / (1 + e−1.10) = 1 / (1 + 0.333) = 0.75.

  3. One email scores 10 and another scores 12. Does the filter think the second is much more likely to be spam?

    Answer

    Not in probability: σ(10) ≈ 0.99995 and σ(12) ≈ 0.99999. Both sit on the flat top of the curve. The odds differ by a factor of e2 ≈ 7.4, but both probabilities round to 1.

Go further

Optional background, for readers who want more than this page.

Where this shows up

Units that build on this one