Concept units

A point in space is a list of numbers, and a vector is an arrow you can add to another or stretch.

Needs: nothing first

The straight-line distance between two points comes from Pythagoras' theorem, in 2D or 3D.

Needs: Vectors: arrows made of numbers

A matrix is a table of numbers that turns one vector into another through weighted sums.

Needs: The dot product: how much two vectors agree

A rotation matrix turns every point of a shape by the same angle without changing its size.

Needs: Multiplying a matrix by a vector, Sine waves

The derivative measures how fast a quantity changes: the slope of its graph at one point.

Needs: nothing first

A random draw picks a value by chance, and a seed makes a computer's random choices repeatable.

Needs: nothing first

The normal distribution describes values that cluster around an average and thin out evenly on both sides.

Needs: Random draws: chance you can repeat, Summarizing a list of numbers

The mean gives a list's typical value, and the standard deviation its typical distance from that value.

Needs: nothing first

Normalizing shifts data to center on zero and rescales it, so different examples share one size.

Needs: Summarizing a list of numbers, Vectors: arrows made of numbers, Distance between two points

Sine and cosine turn an angle or a position into a smooth wave that repeats.

Needs: nothing first

An exponential grows by repeated multiplication, and the logarithm counts the multiplications back.

Needs: nothing first

A graph is a set of things, called nodes, with links, called edges, between some pairs of them.

Needs: nothing first

A connected component is a group of nodes you can travel between along edges; cutting an edge can split it in two.

Needs: Graphs: dots joined by lines

Just enough Python to read the code here (variables, lists, loops, functions), with pointers to full courses.

Needs: nothing first

Top-K accuracy asks whether the right answer appears among the first K entries of a ranked list.

Needs: nothing first

The dot product multiplies two vectors entry by entry and adds the results; it is large when they point the same way.

Needs: Vectors: arrows made of numbers, Distance between two points, Sine waves

The chain rule finds the slope of a chain of steps by multiplying the slopes of the steps.

Needs: The derivative: slope at a point

The gradient collects one slope per input into a vector that points uphill fastest.

Needs: The derivative: slope at a point, Vectors: arrows made of numbers

A NumPy array holds many numbers in a grid with a shape, so one line of code can work on all of them at once.

Needs: Reading Python, Vectors: arrows made of numbers

A PyTorch tensor is an array that can live on a graphics card, where thousands of numbers are processed at once.

Needs: Arrays in NumPy

PyTorch records the operations that produced a number and works backward to find how each input changes it.

Needs: PyTorch tensors on the GPU, The chain rule: slopes of chained steps

Finding the closest point to a query by checking every point works, but it slows down as the points multiply.

Needs: Distance between two points

A KD-tree splits space into boxes, so a search can skip every box that cannot hold a closer point.

Needs: Nearest-neighbor search

Farthest-point sampling covers a shape evenly with a few points, always taking the point whose nearest picked point is farthest away.

Needs: Distance between two points, Shapes as point clouds

The assignment problem pairs each item of one set with one item of another so that the total cost is as small as possible.

Needs: Distance between two points

A point cloud describes a shape by many points on or inside it, in no particular order.

Needs: Vectors: arrows made of numbers, Arrays in NumPy

A 3D image is a grid of tiny boxes called voxels, which need not be the same size along every axis.

Needs: Vectors: arrows made of numbers, Distance between two points, Arrays in NumPy

Segmentation gives every voxel a label number that says which object it belongs to.

Needs: 3D images made of voxels

Supervoxels are small, confident pieces of a segmentation that are later joined into whole objects.

Needs: Segmentation: a label for every voxel

Connectomics reconstructs neurons and their connections from electron-microscope images.

Needs: Segmentation: a label for every voxel, Shapes as point clouds

A merge error joins two objects under one label, and a split error cuts one object into several.

Needs: Segmentation: a label for every voxel, Supervoxels: small pieces to merge

Proofreading finds and fixes merge and split errors, by hand or with software that proposes fixes.

Needs: Two kinds of segmentation error, Supervoxels: small pieces to merge

Data augmentation trains on randomly rotated or jittered copies of each example, so a model learns what should not matter.

Needs: Rotating points with a matrix, Random draws: chance you can repeat, Shapes as point clouds, The bell curve (normal distribution)

A linear layer turns an input vector into an output vector through weighted sums whose weights are learned.

Needs: Multiplying a matrix by a vector

An activation function bends each output of a layer, which lets stacked layers describe curved relationships.

Needs: A linear layer: weighted sums

An MLP stacks linear layers with activation functions between them; it is the basic building block of neural networks.

Needs: A linear layer: weighted sums, Activation functions: bending straight lines

A loss function scores a model's predictions with one number that training tries to make small.

Needs: Distance between two points, Summarizing a list of numbers

Gradient descent improves a model by repeatedly moving its weights a small step downhill on the loss.

Needs: The gradient: the uphill direction, Loss: one number for how wrong

AdamW sizes each weight's step by how its gradients behaved recently, and gently shrinks weights toward zero.

Needs: Gradient descent

Gradient clipping caps the length of the gradient, so one unusual batch cannot throw the weights far off.

Needs: Gradient descent, Reading Python

A learning-rate schedule starts with small steps, grows them, then shrinks them as training settles.

Needs: Gradient descent, Reading Python

Training estimates the loss from a small random batch of examples at each step, which is much faster than using them all.

Needs: Gradient descent

A data loader uses worker processes to prepare the next batches while the GPU trains on the current one.

Needs: Minibatches: a few examples per step, Reading Python

A validation set is data the model never trains on, used to check that it has learned rather than memorized.

Needs: Loss: one number for how wrong, Gradient descent

A model overfits when it fits the noise in its training examples and does worse on new ones; a penalty on large weights is a common remedy.

Needs: Held-out data: the validation set, Gradient descent

An exponential moving average keeps a smoothed copy of the weights, which often gives better results than the latest weights.

Needs: Gradient descent, Summarizing a list of numbers, Minibatches: a few examples per step

A checkpoint saves the weights, the optimizer state and the step count, so training can stop and resume where it left off.

Needs: Minibatches: a few examples per step, Reading Python

Mixed precision does most arithmetic in 16-bit numbers, which roughly halves the memory for activations and speeds up GPUs built for it, at a small cost in accuracy.

Needs: PyTorch tensors on the GPU, Gradient descent, Backpropagation: gradients by the chain rule

A training loop repeats one cycle (get a batch, compute the loss, take a step) and now and then evaluates and saves.

Needs: Minibatches: a few examples per step, Gradient descent, Data loaders: preparing examples in parallel, Checkpoints: saving training to resume it, Held-out data: the validation set, A moving average of the weights (EMA), Mixed precision: 16-bit numbers (bf16), Backpropagation: gradients by the chain rule, Automatic gradients in PyTorch, Gradient clipping, AdamW: steps that adapt, Learning-rate schedules: warm-up, decay

The sigmoid squashes any number into a probability between 0 and 1.

Needs: Exponentials, undone by logarithms

Binary cross-entropy charges a model heavily for confident wrong yes-or-no answers and little for confident right ones.

Needs: The sigmoid: from score to probability, Loss: one number for how wrong

Softmax turns a list of scores into probabilities that are all positive and add up to 1.

Needs: Exponentials, undone by logarithms

Cross-entropy is the loss for choosing one class among many: minus the logarithm of the probability given to the right one.

Needs: Softmax: probabilities over many choices, Loss: one number for how wrong

An embedding represents each item, such as a word or a point, as a learned vector that a network can compute with.

Needs: Vectors: arrows made of numbers, A linear layer: weighted sums

A residual connection adds a layer's input to its output, so a deep network learns small corrections and trains stably.

Needs: The multilayer perceptron (MLP), The chain rule: slopes of chained steps

Layer normalization rescales each token's numbers to a standard mean and spread before the next layer uses them.

Needs: Normalizing data, A linear layer: weighted sums

A generative model learns what a collection of examples looks like, so it can produce new examples that fit in.

Needs: Random draws: chance you can repeat, The bell curve (normal distribution)

A conditional generative model takes extra input, such as part of a shape, and generates outputs that fit it.

Needs: Generative models: making new examples

Shape completion generates the missing part of a partial shape, with the known part as the condition.

Needs: Conditional generation: generating from a hint, Shapes as point clouds

A velocity field gives every point of space, at every moment, an arrow saying where something there moves next.

Needs: Vectors: arrows made of numbers, The derivative: slope at a point

Euler's method follows a velocity field by many small straight steps, each along the arrow where you stand.

Needs: Velocity fields: an arrow at every point, Exponentials, undone by logarithms

Broadcasting lets NumPy combine arrays of different shapes by stretching each array along any axis that it lacks or that has length 1, without copying it.

Needs: Arrays in NumPy

Backpropagation computes every weight's gradient in one backward pass, applying the chain rule layer by layer.

Needs: The chain rule: slopes of chained steps, The multilayer perceptron (MLP), Loss: one number for how wrong

A permutation-invariant function gives the same answer however its inputs are ordered, which a model of point sets needs.

Needs: Shapes as point clouds

A computer stores a decimal number as a sign, a power of two and a fixed number of binary digits, so most numbers are rounded to the nearest stored value, and the gaps between stored values grow with their size.

Needs: Exponentials, undone by logarithms, Arrays in NumPy

A recursive function answers a small case directly and passes the rest of the problem to a new call of itself; the calls stop only if every chain of them reaches that small case.

Needs: Reading Python

Data-parallel training gives each GPU its own share of a batch and averages their gradients before every step.

Needs: Minibatches: a few examples per step, PyTorch tensors on the GPU

Fourier features describe a position by sine waves of several frequencies, which helps a network represent fine detail.

Needs: Sine waves, Embeddings: things as vectors

Attention lets each token gather information from the others, weighting each by how well its key matches the token's query.

Needs: Embeddings: things as vectors, Softmax: probabilities over many choices, Multiplying a matrix by a vector, The dot product: how much two vectors agree

Cross-attention lets one set of tokens send its queries to a second set, such as the tokens of a condition.

Needs: Attention: queries, keys, values

A transformer block combines attention with an MLP, each wrapped in normalization and a residual connection.

Needs: Attention: queries, keys, values, The multilayer perceptron (MLP), Residual connections, Layer normalization

PointNet applies the same MLP to every point and then takes a maximum, so the result does not depend on the points' order.

Needs: The multilayer perceptron (MLP), Order doesn't matter: permutation invariance

Flow matching trains a network to predict the velocity that carries random noise along straight paths to real examples.

Needs: Generative models: making new examples, Euler's method: small steps along a field, The bell curve (normal distribution), Loss: one number for how wrong, Gradient descent, The multilayer perceptron (MLP)

A network that removes noise must know how noisy its input is; adaptive layer normalization (adaLN) feeds it the time.

Needs: Layer normalization, Fourier features: positions as waves, The multilayer perceptron (MLP)

A diffusion transformer is a transformer, told the time in every block, that predicts how to move noisy tokens toward data.

Needs: The transformer block, Flow matching: turning noise into shapes, Time conditioning: telling a network the time

Classifier-free guidance strengthens a condition's effect by extrapolating from an unconditional prediction past a conditional one.

Needs: Conditional generation: generating from a hint, Flow matching: turning noise into shapes

The standard noising process of a diffusion model shrinks an example and adds bell-curve noise over many steps, and any step can be reached in one jump.

Needs: Generative models: making new examples, The bell curve (normal distribution)