Shape completion: generating the missing part

A completion continues a partial shape

A 3D scanner sees a mug from one side, so the scan has no points on the back. A branch runs off the edge of an image, so its end is cut off. In both cases we have a partial shape, stored as a point cloud, and we want a plausible rest. Shape completion is conditional generation where the condition is the known part.

Training pairs come from cutting complete shapes

To train a completion model, you need pairs: a known part and its true missing part. Complete shapes supply them. Cut a complete shape in two, give the model one part as the condition, and train it to generate the other. Each shape gives many pairs, because each cut can fall in a different place. No person has to label anything, because the data supply their own answers. This is called self-supervised learning.

The known part and the missing part stay in the same coordinates, so the generated points land in the right place. Many pipelines also shift and scale each pair so that the known part is centered (see normalizing data).

make_pairs.py

import numpy as np

rng = np.random.default_rng(2)

# A full shape: 12 points along a quarter circle of radius 4.
# np.linspace gives 12 evenly spaced angles from 0 to π/2 (np.pi is π).
a = np.linspace(0, np.pi / 2, 12)
# np.stack(..., axis=1) puts x and y side by side: one row per point.
shape = np.stack([4 * np.cos(a), 4 * np.sin(a)], axis=1)

def make_pair(shape):
    # Cut a full shape at a random x: the right part is known, the left part is missing.
    cut = rng.uniform(1.0, 3.0)             # a random number between 1 and 3
    known = shape[shape[:, 0] > cut]        # the rows whose x is greater than cut
    missing = shape[shape[:, 0] <= cut]     # the rows whose x is at most cut
    return cut, known, missing

for i in range(3):                    # one shape gives many training pairs
    cut, known, missing = make_pair(shape)
    print(f"cut at x = {cut:.2f}: {len(known)} known, {len(missing)} missing")

cut at x = 1.52: 9 known, 3 missing
cut at x = 1.60: 9 known, 3 missing
cut at x = 2.63: 6 known, 6 missing

A generative model keeps the possibilities apart

If a shape may continue upward or downward, a predictor trained with squared error returns the average of the two (see loss): a short stub between them, or points smeared over both. A generative model picks one possibility per sample. Ask for several samples, and you see the range of plausible completions.

The spread of the samples is useful in itself. If many completions agree, the known part settles the answer. If they spread out, the known part leaves the answer open there. Completions can also test a candidate: if a piece of shape lies where many completions go, it plausibly continues the known part.

The output is a set of points, not a list

A point cloud has no order. The same points listed in another order are the same shape (see permutation invariance). So a completion model must produce the missing points as a set. A score for how well a completion matches the true missing part must compare sets, not lists: for example, it can match each generated point to the nearest true point.

Models such as the diffusion transformer treat every point alike, which suits sets. With flow matching, each output point starts as a noise point and moves into place. Its training pairs each noise point with one true point and compares velocities point by point, so its loss needs no comparison of sets.