Proofreading a segmentation
Read these first if they are new to you
- Two kinds of segmentation error Intermediate
- Supervoxels: small pieces to merge Intermediate
The idea in one paragraph
Proofreading is the work of finding and fixing the merge and split errors in a segmentation. A person looks at each object and its image, joins labels to fix a split, and cuts a label to fix a merge. Finding errors takes most of the time, because there are far too many objects to check everything by eye. So software proposes likely fixes, each with a score, and a person answers yes or no. Fixes with very high scores can be applied without a person, at the risk of adding some new errors.
By hand, a proofreader looks for shapes that cannot be right
A segmentation made by a program has errors of two kinds (see two kinds of segmentation error). A proofreader is a person who fixes them. They use a viewer that shows each object as a 3D surface, next to the image slices it came from. They follow an object along its length and look for clues:
- A sign of a split: the object stops suddenly, in the middle of the image, with no natural end. The proofreader looks nearby for a piece that continues it.
- A sign of a merge: the shape is impossible, such as two tubes crossing in an X, or a sudden jump in thickness. The proofreader checks the image slices for a membrane the program missed.
The viewer offers two edits. A join gives two labels one label; it fixes a split. A cut divides one label in two; it fixes a merge. To cut, the proofreader marks a point on each side, and the software divides the object's supervoxels between the two sides. Because both edits change the lookup table rather than the voxels, either can be undone.
Software proposes fixes, and people decide
A large image can hold many thousands of objects, so checking every one by eye takes years of work. Software therefore does the searching. For split errors, a common plan has four steps:
- Find the ends of objects, where a split might be.
- For each end, find pieces close by (see nearest-neighbor search).
- Give each (end, nearby piece) pair a score from 0 to 1 for how likely they belong together. A trained model computes it from the image or from the shapes; one way is to predict how the object would continue (see shape completion) and check whether the nearby piece matches.
- Sort the pairs by score and show them to a person, highest first. The person answers yes or no.
A score threshold trades people's time against new errors
You can apply every proposal whose score is at or above a threshold without asking anyone. A high threshold applies few fixes, nearly all right. A lower one applies more fixes, but some are wrong, and each wrong join is a new merge error.
A model scores twelve proposed joins: 0.97, 0.95, 0.93, 0.91, 0.84, 0.78, 0.71, 0.62, 0.45, 0.30, 0.22 and 0.10. Later checking shows that seven are right: all five above 0.8, plus 0.71 and 0.62.
- Threshold 0.9: four joins are applied (0.97 to 0.91). All four are right. Eight are left for people.
- Threshold 0.7: seven joins are applied (0.97 to 0.71). Six are right, and 0.78 is wrong, so one new merge error appears. Five are left for people.
Lowering the threshold saved three decisions but made one merge error, which costs a cut to fix later.
Here the lowest safe threshold is 0.79. It applies five right joins and leaves seven for people. In practice nobody knows which proposals are wrong in advance, so teams measure this on a sample that people have checked, then pick the threshold.
proofread.py
import numpy as np
# Twelve proposed joins: the model's score, and whether the join is right (1) or not (0)
score = np.array([0.97, 0.95, 0.93, 0.91, 0.84, 0.78,
0.71, 0.62, 0.45, 0.30, 0.22, 0.10])
right = np.array([1, 1, 1, 1, 1, 0, 1, 1, 0, 0, 0, 0], dtype=bool)
for threshold in (0.9, 0.7):
auto = score >= threshold # True for each join applied without a person
# & means "and", ~ means "not", and .sum() counts the True values
good, bad = (auto & right).sum(), (auto & ~right).sum()
print(threshold, "correct:", good, " wrong:", bad, " left for people:", (~auto).sum())
0.9 correct: 4 wrong: 0 left for people: 8
0.7 correct: 6 wrong: 1 left for people: 5Real proofreading differs between teams. A score is not always a true probability, so 0.9 need not mean "right 9 times in 10." People make mistakes too, so important objects are often checked by a second person. Proposals to cut merges also exist, but they are harder to make, because the software must also say where to cut.
Check yourself
-
While following an object, you see it stop abruptly in the middle of the image. Which kind of error does that suggest, and which edit fixes it?
Answer
A split error: the object probably continues in another piece. Find that piece and join it to the object.
-
In the worked example, what happens at threshold 0.6?
Answer
Eight joins are applied (0.97 to 0.62). Seven are right and one (0.78) is wrong, so one new merge error appears. Four proposals are left for people.
-
Why not apply every proposal that scores above 0.5 and skip people altogether?
Answer
Some of those joins are wrong, and each wrong join is a new merge error. Merge errors are costly: they spoil every measurement on the object, and someone must later find them and cut in the right place.
Go further
Optional background. This unit covers what you need here.
-
A worked system: classifiers propose merges and splits, people answer yes or no, and a threshold runs a fully automatic mode.
-
Try proofreading yourself: a citizen-science game where you add missing pieces to a cell in a real image.
-
A widely used web viewer for large 3D images and their segmentations; its README links live examples.
Where this shows up
No unit or explanation builds on this one yet.