Skip to the content ForoHackers
Log inJoin

A benchmark score is a measurement. Where are the error bars?

AI evaluationData scienceProbability and statistics
#1086 · overfit_otter · · Report

A familiar table: model A scores 84%, model B scores 86%, B wins. The test set has 500 examples.

A quick check. The standard error of an accuracy near 0.85 on 500 examples is sqrt(0.85 * 0.15 / 500), about 0.016. A 95% interval is roughly ±3 points. A 2 point difference is inside the noise of each number.

Better than comparing two intervals: compare the models on the same examples, since both are scored against the same items (a paired comparison). A bootstrap on the per-example difference is hard to get wrong:

import random

def boot_diff(a_correct, b_correct, n_boot=10000, seed=0):
    rng = random.Random(seed)
    n = len(a_correct)
    diffs = []
    for _ in range(n_boot):
        idx = [rng.randrange(n) for _ in range(n)]
        diffs.append(sum(b_correct[i] - a_correct[i] for i in idx) / n)
    diffs.sort()
    return diffs[int(0.025 * n_boot)], diffs[int(0.975 * n_boot)]

a_correct and b_correct are lists of 0 and 1, one entry per example. If the interval contains 0, you have not shown that B is better.

My other two rules: always include a trivial baseline (majority class, a keyword rule, last value) in the table, and report the test set size next to every number. A score without n is a rumour with decimals.

#1690 · gradient_descent_hiker · · Report

Agree. I have been guilty of reporting the best of five seeds. If the five runs differ by more than the gap between two methods, the gap says nothing. I now report mean and spread over seeds, and the same split for every method.

#1691 · git_blame_myself · · Report

And the other source of optimism: if you tune hyperparameters on the test set, the "test" number stops being a test. I did that. See my leakage post-mortem. Keep a held-out set that nothing touches until the very end.

#1692 · two_phase_commitment · · Report

Small point on the bootstrap: resample the examples, as you do, not the predictions of each model independently, otherwise you throw away the pairing and your interval gets wider than it should. Yours is correct.

#1693 · overfit_otter · · Report

Right, resampling indices once and applying them to both is what keeps the pairing. If the two models fail on the same examples, the difference is far less noisy than the two scores suggest. That is the good news.

#1697 · bayes_and_bagels · · Report

A Bayesian reading of the same table: instead of "is 0 inside the interval", ask "what is the probability that B is better than A given this data". On 500 examples and a 2 point gap, that probability is nowhere near the 99% people hear when they read "B wins". Cheerful reminder that the gap is a hypothesis, not a fact.

Log in or join to reply.