Segmentation: a label for every voxel
Read these first if they are new to you
- 3D images made of voxels Intermediate
The idea in one paragraph
A segmentation of a 3D image is a second grid of the same size. Each of its voxels holds a whole number, a label, that says which object that voxel belongs to. All voxels with the same label form one object. Label 0 usually means "no object." The labels are names, not amounts: label 7 is not bigger than label 5. From a segmentation you can count each object's voxels, find its volume and center, and pull it out on its own.
Each voxel gets one label number
Look at a microscope image of tissue. You see cells as light regions with dark edges: their membranes, the thin outer skin of each cell. You can tell where one cell ends and the next begins, but the computer only has brightness numbers (see 3D images made of voxels). It does not know which voxels belong together.
A segmentation records that grouping. It is a grid of the same size as the image. In each voxel it stores a label, a whole number. Voxels that share a label make up one object, such as one cell. Voxels between objects get label 0, the background.
Before you use the figure, predict: how many voxels does object 2 have in this slice?
Object 2 has 14 voxels in this slice. In 3D, an object continues into the slices above and below, and its voxels with the same label there belong to the same object.
A label is a name, not an amount
Label numbers only tell objects apart. You can swap labels 2 and 3 everywhere and nothing about the objects changes. Adding or averaging labels means nothing. This is also why a display can pick any color for any label.
Each voxel holds exactly one label. So objects never overlap: a voxel cannot belong to two objects at once.
A semantic segmentation labels what kind of thing each voxel is: every cell gets label 1 and every blood vessel label 2. An instance segmentation gives each separate object its own label, as in the figure. To trace individual cells, you need instance segmentation.
Counting voxels gives sizes and positions
Once each voxel has a label, simple counting answers questions about the objects.
Take the slice in the figure as a 3D image one voxel thick, with voxels 8 × 8 × 40 nm.
- Count the voxels with label 1: 3 + 3 + 2 + 1 = 9, one count per row.
- One voxel has a volume of 8 × 8 × 40 = 2,560 nm³.
- So object 1 has a volume of 9 × 2,560 = 23,040 nm³.
- The center of object 3 is the mean of its voxels' row and column indices. Its 12 voxels lie in rows 3, 4 and 5 (2, 4 and 6 voxels), so the mean row is (2 × 3 + 4 × 4 + 6 × 5) ÷ 12 = 52 ÷ 12 ≈ 4.33.
To turn the center into nanometers, multiply each index by the voxel size along its axis.
You can also pull one object out. The positions of all voxels with label 3 form a list of points, which is a point cloud of that object.
segmentation.py
import numpy as np
seg = np.array([ # one slice of a label array; 0 means "no object"
[1, 1, 1, 0, 2, 2, 2, 2],
[1, 1, 1, 0, 2, 2, 2, 2],
[1, 1, 0, 0, 0, 2, 2, 2],
[1, 0, 3, 3, 0, 0, 2, 2],
[0, 3, 3, 3, 3, 0, 0, 2],
[3, 3, 3, 3, 3, 3, 0, 0],
])
# np.unique lists each label once, and counts the voxels that have it.
labels, counts = np.unique(seg, return_counts=True)
print(labels, counts)
voxel_volume = 8 * 8 * 40 # nm^3 for an 8 x 8 x 40 nm voxel
print(counts[labels == 1] * voxel_volume) # labels == 1 picks object 1's count
rows, cols = np.nonzero(seg == 3) # the row and column of each voxel of object 3
print(round(rows.mean(), 2), round(cols.mean(), 2)) # its center, in voxel units
[0 1 2 3] [13 9 14 12]
[23040]
4.33 2.5Programs store labels as 64-bit whole numbers, so that a huge volume never runs out of label numbers. The arrays are also huge, so programs keep them in many small blocks on disk. Not every tool uses 0 for background, either. And real segmentations are made by programs, often a neural network that marks the membranes between cells followed by a step that groups the voxels inside them. See supervoxels for that grouping step.
Check yourself
-
Two neighboring objects have labels 5 and 7. A friend says object 7 must be the larger one. Are they right?
Answer
No. Labels are names. Only counting the voxels with each label tells you which object is larger.
-
In the figure, what is the volume of object 2, with voxels 8 × 8 × 40 nm?
Answer
Object 2 has 4 + 4 + 3 + 2 + 1 = 14 voxels, so its volume is 14 × 2,560 = 35,840 nm³.
-
Can one voxel belong to both object 1 and object 3 in this kind of segmentation?
Answer
No. Each voxel stores one label, so it belongs to exactly one object, or to the background.
Go further
Optional background. This unit covers what you need here.
-
A short, runnable example that turns a 2D image into a label array and draws a box around each object.
-
Names the semantic and instance kinds; the rest of the article surveys the many ways to compute a segmentation.
Where this shows up
Units that build on this one
- Supervoxels: small pieces to merge Intermediate
- Connectomics: mapping neurons from images Intermediate
- Two kinds of segmentation error Intermediate