Top-K accuracy

A case counts as a hit when the right answer is in the first K

Picture an app that names a bird from a photo. It shows its six best guesses, most likely first. If the right bird is first, the app was right outright. If the right bird is third, a birdwatcher who reads the top three still finds it. Top-K accuracy measures how often a reader who checks the first K guesses finds the right one.

To compute it, take a set of test cases where you know the right answer. For each case:

1. Find the rank of the right answer in the list: 1 if it is first, 2 if it is second, and so on.
2. Call the case a hit if the rank is K or less.

Top-K accuracy is the number of hits divided by the number of cases.

Predict first. In the five cases below, the right answer has ranks 1, 3, 2, 1 and 6. Guess the top-1 and top-3 accuracy before you read on.

Pick K to match how many guesses someone will check

Top-K accuracy is useful when a person, or a later step in a program, looks at a short list rather than one answer. Then the honest question is "how often is the right answer on the list they will see?" A bigger K finds more right answers, but someone must check more candidates per case.

Compare against chance. If a system put N candidates in random order, the right answer would land in the first K with probability K / N. With 100 candidates, random order gives a top-10 accuracy of about 10%. A useful system beats that by a wide margin.

Readers who know a little Python (see Reading Python) can compute ranks from a system's scores. The rank of the right answer is 1 plus the number of candidates that score higher.

topk.py

import numpy as np

# Each row holds one case's scores for 6 candidates.
scores = np.array([[0.9, 0.1, 0.3, 0.2, 0.0, 0.4],
                   [0.2, 0.8, 0.1, 0.7, 0.3, 0.6],
                   [0.1, 0.3, 0.2, 0.9, 0.4, 0.0]])
right = np.array([0, 5, 4])   # the index of the right candidate

# Rank = 1 + how many candidates score higher than the right one.
# np.arange(len(right)) gives the row numbers [0, 1, 2]. Paired with right,
# they pick scores[0, 0], scores[1, 5] and scores[2, 4]: each case's right score.
right_score = scores[np.arange(len(right)), right]
# right_score[:, None] stands the 3 scores in a column, so each row is compared
# with its own right score. sum(axis=1) counts the True results in each row.
rank = 1 + (scores > right_score[:, None]).sum(axis=1)
print(rank)
for k in (1, 2, 3):
    # rank <= k means "rank at most k". The mean counts True as 1 and False as 0.
    print(k, round((rank <= k).mean(), 2))

[1 3 2]
1 0.33
2 0.67
3 1.0