Generative models: making new examples

The examples are random draws from a distribution nobody wrote down

Think of every handwritten digit 7 ever written. No two are the same, but together they have a typical shape and a typical amount of variety. A distribution describes how likely each possible example is (see random draws). A collection of examples is a set of random draws from such a distribution. We see the examples, but never the distribution itself.

A generative model has two jobs. It learns the distribution from the examples. Then it draws new examples from what it learned. Drawing an example is called sampling, and the example drawn is a sample.

The simplest generative model fits a bell curve and draws from it

Some data are spread like a bell curve, such as the heights of adults. For such data, a generative model needs only two numbers: the mean and the spread.

Your turn. With the same model, what height does the draw z = −2 give?

One bell curve cannot draw a ring, but bent noise can

Now let the examples be points on a ring of radius 1 in a flat plane. Fit a bell curve in two dimensions, with a mean and a spread for each coordinate. Its center is the average point, (0, 0), the middle of the ring, where there are no examples at all.

The fix is to keep the noise simple and make the function more flexible. The second model below draws the same kind of 2D noise, then bends it. It keeps each noise point's direction from the center, and moves the point out to a distance near 1.

A good model's samples are new, and spread like the data

Two simple models fail in opposite ways.

• A copier returns one of its training examples at random. Each sample looks perfect, but it is never new.
• An averager returns the average example. For the ring, that is (0, 0), a point with no example near it. For photos of faces, it is a blur.

A good model's samples are new, and as a group they are spread like the real examples: the same typical shape and the same variety. So you judge a generative model by many samples at once, not by one sample against one example.

Several families of generative models exist. They differ in how they learn the function. Diffusion and flow-matching models turn noise into an example over many small steps. Generative adversarial networks (GANs) train the function against a second network that tries to tell samples from real examples. Variational autoencoders (VAEs) learn to squeeze each example into a few numbers and to rebuild it. Autoregressive models build an example one piece at a time, such as one word after another.

bell_curve_model.py

import numpy as np

rng = np.random.default_rng(0)

# The simplest generative model: fit a bell curve, then draw from it.
heights = np.array([160.0, 165.0, 170.0, 175.0, 180.0])
mu, sigma = heights.mean(), heights.std()
print("mean", mu, "spread", round(sigma, 2))
z = rng.standard_normal(3)                 # three noise draws
print("new heights", np.round(mu + sigma * z, 1))

# Ring-shaped data in 2D: one bell curve puts samples in the empty middle.
angle = rng.uniform(0, 2 * np.pi, 1000)    # 1000 random angles from 0 to 2π
ring = np.stack([np.cos(angle), np.sin(angle)], axis=1)   # one row (cos, sin) per angle
# Fit one 2D bell curve to the ring, then draw 1000 points from it.
fake = rng.multivariate_normal(ring.mean(axis=0), np.cov(ring.T), 1000)
# np.linalg.norm(..., axis=1) gives each point's distance from the center.
print("bell curve, share within 0.5 of center:", np.mean(np.linalg.norm(fake, axis=1) < 0.5))

# A bent bell curve: a fixed function turns each 2D noise draw into a ring point.
z2 = rng.standard_normal((1000, 2))
r = np.linalg.norm(z2, axis=1, keepdims=True)    # each draw's distance, as a column
bent = z2 / r * (1 + 0.15 * (r - 1.25))          # same direction, distance near 1
print("bent noise, share within 0.5 of center:", np.mean(np.linalg.norm(bent, axis=1) < 0.5))

mean 170.0 spread 7.07
new heights [170.9 169.1 174.5]
bell curve, share within 0.5 of center: 0.207
bent noise, share within 0.5 of center: 0.0

About one bell-curve sample in five lands in the empty middle. No bent sample does.