Two kinds of segmentation error
Read these first if they are new to you
- Segmentation: a label for every voxel Intermediate
- Supervoxels: small pieces to merge Intermediate
The idea in one paragraph
A segmentation can be wrong in two ways. A merge error gives one label to parts of two or more real objects. A split error spreads one real object over two or more labels. A merge needs a cut to fix, and a split needs a join. To count the errors, compare each label with the true objects it covers, and each true object with the labels it is spread over.
A merge joins too much; a split joins too little
Imagine two thin cells that run side by side and touch at one spot. A correct segmentation gives each cell its own label. The figure shows that, and the two ways it can go wrong. Each cell is drawn in three pieces, its supervoxels, and pieces with one color share one label.
Truth: two objects
Top and bottom have different labels.
Merge error: one label for both
Joined where they touch.
Split error: one object in two labels
The top object is cut between pieces 1 and 2.
A merge error happens when one label covers parts of two or more true objects. It usually comes from a wrong join, often where two objects touch, or from a supervoxel that spans two objects. A split error happens when one true object is spread over two or more labels. It usually comes from a missing join, for example where the image is blurry and a faint smudge looks like a membrane cutting through an object.
Count errors by comparing labels with true objects
Suppose you know the true objects, from careful hand work. Then you can count both kinds of error with two rules:
- For each label, count the true objects it covers. Every one beyond the first is a merge error.
- For each true object, count the labels it is spread over. Every one beyond the first is a split error.
Each merge error needs one cut to fix. Each split error needs one join.
The truth is object A = {1, 2, 3} and object B = {4, 5, 6}. A program gives two labels: {1} and {2, 3, 4, 5, 6}.
- Label {1} covers only A: no merge. Label {2, 3, 4, 5, 6} covers A and B: one merge error.
- Object A is spread over two labels, {1} and {2, 3, 4, 5, 6}: one split error. Object B lies in one label: no split.
- To fix it, cut between 2 and 5, where the tubes touch. Then join 1 and 2.
The program has the right number of labels, two, but both are wrong. Counting labels alone hides errors.
Before you use the figure, predict the counts for each choice. Then check.
The same rules in NumPy: list which (true object, label) pairs occur, then count per side.
seg_errors.py
import numpy as np
truth = np.array([1, 1, 1, 2, 2, 2]) # true neuron of supervoxels 1..6
pred = np.array([10, 20, 20, 20, 20, 20]) # object the segmentation gave each
pairs = np.unique(np.stack([truth, pred], axis=1), axis=0) # which (true, given) pairs occur
print(pairs)
# For each true object t: count the different labels in rows whose column 0 is t, minus 1.
splits = sum([len(np.unique(pairs[pairs[:, 0] == t, 1])) - 1 for t in np.unique(truth)])
# For each label p: count the different true objects in rows whose column 1 is p, minus 1.
merges = sum([len(np.unique(pairs[pairs[:, 1] == p, 0])) - 1 for p in np.unique(pred)])
print("split errors:", splits, " merge errors:", merges)
[[ 1 10]
[ 1 20]
[ 2 20]]
split errors: 1 merge errors: 1Merge errors usually cost more than split errors
A split leaves a piece of an object missing. The rest of the object is still correct, and a join restores it. A merge makes an object that does not exist, and everything measured on it, such as its size, shape and contacts, is wrong. Fixing a merge also means finding the right place to cut. For these reasons, programs are usually tuned to make more splits and fewer merges.
The counts here treat every error alike: a tiny piece cut off counts the same as an object cut in half. Real evaluations weigh errors by size, for example by the number of voxels or the length of object affected, and use scores such as variation of information. They also assume the "truth" is right, though it was made by people and has errors of its own.
Check yourself
-
The truth is A = {1, 2, 3} and B = {4, 5, 6}. A program gives {1, 2, 3, 4} and {5, 6}. Count the merge and split errors.
Answer
Label {1, 2, 3, 4} covers A and B: one merge error. Object B is spread over {1, 2, 3, 4} and {5, 6}: one split error. Object A lies in one label: no split. So one merge and one split.
-
A program gives every supervoxel its own label: {1}, {2}, … {6}. How many errors of each kind?
Answer
No label covers two objects, so there are no merge errors. Each true object is spread over 3 labels, which is 2 split errors each, so 4 split errors in all.
-
Why do segmentation programs usually prefer a split error to a merge error?
Answer
A split is fixed by one join, which is easy and safe, and the rest of the object is still correct. A merge creates an object that does not exist, spoils every measurement on it, and needs a cut placed in the right spot.
Go further
Optional background. This unit covers what you need here.
-
The introduction defines merge and split errors on real brain images, and notes that finding errors takes more of a person's time than fixing them.
-
A citizen-science game where you add missing pieces to a cell, which lets you see split and merge errors for yourself.
Where this shows up
Units that build on this one
- Proofreading a segmentation Intermediate