Normalizing data
Read these first if they are new to you
The idea in one paragraph
Data often arrive at very different positions and sizes: one list in centimeters, another in kilometers; one shape near the origin, another far away and ten times larger. Normalizing makes them comparable in two steps. First subtract the center, so the data sit around zero. Then divide by a measure of size, so the data have size one. Keep the two numbers you used, because they let you undo the change later.
Subtract the mean, then divide by the spread
Take five heights in centimeters: 150, 160, 170, 180 and 190. Their mean is 170 and their standard deviation is about 14.14, computed as in the mean-and-spread unit. Subtract the mean from each height, then divide by the standard deviation. The result for each value is its z-score: how many standard deviations it lies above or below the mean.
z = (value − mean) / standard deviation
The new list has mean 0 and standard deviation 1, whatever the units were. A height of 190 cm becomes 1.41, and so would a weight or a price that sat the same number of standard deviations above its own mean. This step is called standardizing.
- Mean: (150 + 160 + 170 + 180 + 190) / 5 = 170.
- Deviations: −20, −10, 0, 10, 20. Squares: 400, 100, 0, 100, 400. Their mean is 1000 / 5 = 200, so the standard deviation is √200 ≈ 14.14.
- Divide each deviation by 14.14: −1.41, −0.71, 0, 0.71, 1.41.
Your turn. A list has mean 50 and standard deviation 5. What are the z-scores of 60 and 47.5? What value has a z-score of −2?
Answer
(60 − 50) / 5 = 2 and (47.5 − 50) / 5 = −0.5. To undo, multiply and add back: −2 × 5 + 50 = 40.
For a shape, move its center to the origin and scale it to size one
A shape made of points is a list of vectors. To normalize it, first find its center: the mean point, whose x is the mean of all the x's and whose y is the mean of all the y's. Subtract the center from every point; this is called centering. Then pick one number for the shape's size, such as the distance from the center to the farthest point, and divide every point by it. Afterwards every point lies within distance 1 of the origin.
The triangle with corners (2, 2), (4, 2) and (3, 5).
- Center: ((2 + 4 + 3) / 3, (2 + 2 + 5) / 3) = (3, 3).
- Subtract it: (−1, −1), (1, −1), (0, 2).
- Distances from the origin: √2 ≈ 1.41, √2 ≈ 1.41, and 2. The largest is 2.
- Divide by 2: (−0.5, −0.5), (0.5, −0.5), (0, 1).
The house's peak, the corner farthest from its center, is the one that touches the circle.
There is more than one choice for "center" and "size." The size can be the farthest distance, the standard deviation, or the width of a bounding box; each gives a different scale. Pick one and use it everywhere. Also, a model trained on normalized data expects new data normalized with the same recipe, and its answers come back in normalized units. Store the center and size for each example so you can turn the answers back into the original units.
norm.py: z-scores for a list, and a centered and scaled shape
import numpy as np
heights = np.array([150, 160, 170, 180, 190])
z = (heights - heights.mean()) / heights.std()
print(np.round(z, 2))
shape = np.array([[2.0, 2.0], [4.0, 2.0], [3.0, 5.0]]) # one point per row
center = shape.mean(axis=0)
centered = shape - center
size = np.linalg.norm(centered, axis=1).max() # farthest point from the center
print(center, size)
print(centered / size)
[-1.41 -0.71 0. 0.71 1.41] [3. 3.] 2.0 [[-0.5 -0.5] [ 0.5 -0.5] [ 0. 1. ]]
axis=0 takes the mean down each column, giving one mean per coordinate.
axis=1 takes each point's length across its row.
Check yourself
-
Scores have mean 170 and standard deviation 15. What is the z-score of 185?
Answer
(185 − 170) / 15 = 1: one standard deviation above the mean.
-
A shape was normalized with center (10, 4) and size 5. A program returns the normalized point (0.2, −1). Where is that point in the original units?
Answer
Multiply by the size, then add the center: (0.2 × 5 + 10, −1 × 5 + 4) = (11, −1).
-
You double every point of a shape and then normalize it. How does the result compare with normalizing the original shape?
Answer
It is the same. Doubling doubles the center and the size, and the division cancels it. That is the point of normalizing: size and position no longer matter.
Go further
Optional background, for readers who want more than this page.
-
Z-scores explained in plain language, in its "Standard Scores" section.
-
Standardization, or mean removal and variance scaling scikit-learn user guide Reference Intermediate
The code view: why many learning methods expect standardized inputs, and how to reuse the same recipe on new data.
Where this shows up
Units that build on this one
- Layer normalization Intermediate