Floating-point numbers: rounding to the nearest stored value

A float is scientific notation in base 2

In scientific notation, a decimal number is written as a few digits times a power of ten: 6,500 = 6.5 × 10³. A computer does the same in base 2. In binary, each digit (a bit) is 0 or 1. The place values left of the point are 1, 2, 4, 8 and so on, the powers of two from the exponentials unit. Right of the point they are 1/2, 1/4, 1/8, and so on. We write 2⁻³ for 1/2³ = 1/8. So binary 110.1 is 4 + 2 + 1/2 = 6.5, and 6.5 = 1.101 (binary) × 2².

A floating-point number, or float, stores three pieces of that form. The sign says plus or minus. The exponent is the power of two, here 2. The fraction holds the bits after the point, here 101. The bit before the point is always 1, except for zero and the tiny numbers in the warning box below, so the format does not store it. The number of fraction bits sets how many significant digits a float keeps. The number of exponent bits sets how large and how small it can be.

Each format also has 1 sign bit. These are the float64 and float32 dtypes of NumPy arrays. Both follow the IEEE 754 standard, which almost every computer uses today. On almost every platform, Python's float is float64.

The gap between stored values doubles at each power of two

Take a toy format with only 2 fraction bits. Between 1 and 2 it can store 1.00, 1.01, 1.10 and 1.11 in binary, which are 1, 1.25, 1.5 and 1.75. Between 2 and 4 the exponent goes up by one, so the same four patterns are doubled: 2, 2.5, 3 and 3.5. Each range from one power of two to the next holds the same count of values, so the gap doubles from one range to the next.

A real format works the same way with more bits. With f fraction bits, the values between 2^k and 2^k+1 are spaced 2^k × 2^−f apart:

gap = 2^{k − f}

For float32, f = 23, so the gap between 1 and 2 is 2⁻²³, about 1.2 × 10⁻⁷. For float64, f = 52, and that gap is 2⁻⁵², about 2.2 × 10⁻¹⁶. The gap just above 1 is called the format's machine epsilon.

A number that falls between two stored values is rounded to the nearer one. A number exactly halfway goes to the neighbour whose last bit is 0. So the error is at most half a gap. The gap grows with the number, so the error is at most the same small share of the number. That share is 2⁻²⁴, about 6 × 10⁻⁸, for float32, and 2⁻⁵³, about 1.1 × 10⁻¹⁶, for float64. That share is where the "about 7" and "about 16" significant digits in the table come from.

Most decimals, such as 0.1, have no exact binary form

A binary number with finitely many bits after the point is a whole number divided by a power of two, such as 0.101 (binary) = 5/8. The number 0.1 is 1/10. If 1/10 equalled J/2^N for whole numbers J and N, then 10 × J would equal 2^N. But 10 × J is divisible by 5, and no power of two is. So 0.1 has no exact binary form. Its bits repeat forever, as 1/3 = 0.333… does in decimal, and the computer must round them.

Python stores 0.1 as the nearest float64, which is 0.1000000000000000055511151231257827021181583404541015625. The numbers 0.2 and 0.3 are rounded too. Each addition is then computed exactly and rounded once more. For 0.1 + 0.2, the errors do not cancel: the sum lands on 0.30000000000000004, the stored value just above the one Python uses for 0.3.

The snippet shows this, and one more effect: errors can add up over many operations. In the code, f"{x:.20f}" writes x with 20 digits after the point. Python writes 2.2 × 10⁻¹⁶ as 2.2e-16, in code and in its output. np.float32(x) stores x as a float32, and int turns a float into a whole number for printing. np.spacing(x) gives the gap from x to the next stored value of its type, and np.finfo(...).eps gives machine epsilon.

floats.py

import math
import numpy as np

print(0.1 + 0.2, 0.1 + 0.2 == 0.3)
print(f"{0.1:.20f}")                  # the stored 0.1, to 20 decimals
print(math.isclose(0.1 + 0.2, 0.3))   # equal to within a tolerance?

total = 0.0
for i in range(10):
    total = total + 0.1
print(total)                          # ten roundings add up

count = np.float32(16777216)          # 2**24, as a float32
print(int(count + np.float32(1)))     # the added 1 is lost
print(np.spacing(count), np.spacing(16777216.0))
print(np.finfo(np.float32).eps, np.finfo(np.float64).eps)

0.30000000000000004 False
0.10000000000000000555
True
0.9999999999999999
16777216
2.0 3.725290298461914e-09
1.1920929e-07 2.220446049250313e-16

Compare floats with a tolerance, not with ==

Two calculations that are equal on paper often differ in their last bits. So a test should ask whether two floats are close, not whether they are equal. math.isclose(a, b) returns True when a and b differ by at most a share of the larger one. By default that share, rel_tol, is 10⁻⁹, about 9 matching digits. A tolerance of this kind is called relative, because it grows with the numbers, as the gaps do.

A relative tolerance fails near zero. With the defaults, math.isclose(1e-20, 0) is False. The allowed difference is 10⁻⁹ of the larger value, here 10⁻²⁹, and the actual difference, 10⁻²⁰, is far larger. When one value may be zero, also pass an absolute tolerance, a fixed amount, such as abs_tol=1e-12, chosen for the size of your data. For whole arrays, np.isclose does a similar job, with a slightly different rule and other defaults: rtol=1e-05 and atol=1e-08.