The diffusion noising process: from data to noise

Each step shrinks the example a little and adds a little noise

A diffusion model is a generative model that learns from the opposite direction: a fixed process that turns examples into noise. This is the noising process, also called the forward process. This page follows the DDPM paper (Ho, Jain and Abbeel, 2020), which uses T = 1000 steps. Step t makes x_t from x_t−1:

x_t = √(1 − β_t) × x_t−1 + √β_t × ε_t

Here ε_t (epsilon) is a fresh draw from the standard normal, the bell curve with mean 0 and standard deviation 1. β_t (beta), between 0 and 1, sets the step's size, and the list β₁, …, β_T is the noise schedule. So the step shrinks the value by the factor √(1 − β_t), then adds bell-curve noise with standard deviation √β_t.

The shrink keeps the variance at 1

The reason uses the variance, the square of the standard deviation, and two standard rules of probability:

• Scaling. Multiplying a random value by c multiplies its variance by c².
• Adding. Two random values are independent when knowing one tells you nothing about the other. A sum of independent values has the sum of their variances. If both are normal, so is the sum.

Each noise draw is fresh, so it is independent of the value it joins. If x_t−1 has variance 1, the rules give x_t the variance

(1 − β_t) × 1 + β_t × 1 = 1.

Without the shrink, the variance would grow by β_t at every step, and the example would stay at full size. Song and others (2021) call this form variance preserving. The variance stays exactly 1 only for data with variance 1; other data move toward it.

The noise of many steps adds up to one draw, so any step is one jump away

Put the formula for x₁ into the formula for x₂:

The part in brackets is a sum of two independent normal draws with mean 0. By the rules, it is one normal draw with variance (1 − β₂) β₁ + β₂, which multiplies out to 1 − (1 − β₁)(1 − β₂). The same argument works for any number of steps. Write α_t = 1 − β_t (alpha), and let ᾱ_t (alpha bar) be the product α₁ × α₂ × … × α_t. Then

x_t = √ᾱ_t × x₀ + √(1 − ᾱ_t) × ε,

where ε is one standard normal draw. √ᾱ_t is the signal scale, the factor left on the example. √(1 − ᾱ_t) is the standard deviation of the noise. Every α_t is below 1, so ᾱ_t falls at every step, and the example fades.

Your turn. With the same β values, start from x₀ = −0.5, with the draws ε₁ = 1.0 and ε₂ = −1.0. Find x₂ both ways.

The jump makes training cheap. DDPM picks a random step t from 1 to T, makes x_t in one jump, and trains a network to predict ε from x_t and t. The network needs t to know how noisy its input is (see time conditioning).

noising.py: 500 single steps against one jump, with the DDPM schedule

import numpy as np

beta = np.linspace(0.0001, 0.02, 1000)   # DDPM: beta_1 ... beta_1000
alpha_bar = np.cumprod(1 - beta)         # step t is at index t - 1
a = alpha_bar[499]                       # step 500

rng = np.random.default_rng(0)
x = np.full(100_000, 0.8)                # 100,000 copies of x0
for t in range(1, 501):                  # t = 1, 2, ..., 500
    x = np.sqrt(1 - beta[t - 1]) * x + np.sqrt(beta[t - 1]) * rng.standard_normal(100_000)
jump = np.sqrt(a) * 0.8 + np.sqrt(1 - a) * rng.standard_normal(100_000)

print("500 steps: mean", round(x.mean(), 3), "std", round(x.std(), 3))
print("one jump:  mean", round(jump.mean(), 3), "std", round(jump.std(), 3))
print("formula:   mean", round(np.sqrt(a) * 0.8, 3), "std", round(np.sqrt(1 - a), 3))

500 steps: mean 0.227 std 0.964
one jump:  mean 0.226 std 0.954
formula:   mean 0.224 std 0.96

np.linspace(a, b, n) gives n evenly spaced numbers from a to b, and np.cumprod gives running products. Both ways agree with the formula to within 0.01. The small differences come from using a finite number of draws.

The schedule sets how fast the example fades

The DDPM schedule is linear: β_t rises in a straight line from 0.0001 to 0.02 over 1000 steps. Each β_t is small, but ᾱ_T, a product of 1000 numbers below 1, is about 0.00004. So √ᾱ_T ≈ 0.0064: less than 1% of the example's scale is left.

Nichol and Dhariwal (2021) found that, for images of 64 × 64 and 32 × 32 pixels, the linear schedule destroys the example faster than needed. Their cosine schedule builds ᾱ_t from a cosine curve. It falls slowly at the start and the end, and almost in a straight line in between.

After the last step, every example has become almost pure noise

With the linear schedule, x_T ≈ 0.0064 × x₀ + 1.0 × ε: nearly a standard normal draw, whatever the example was. So generation can start from fresh noise.