A familiar table: model A scores 84%, model B scores 86%, B wins. The test set has 500 examples.
A quick check. The standard error of an accuracy near 0.85 on 500 examples is sqrt(0.85 * 0.15 / 500), about 0.016. A 95% interval is roughly ±3 points. A 2 point difference is inside the noise of each number.
Better than comparing two intervals: compare the models on the same examples, since both are scored against the same items (a paired comparison). A bootstrap on the per-example difference is hard to get wrong:
import random
def boot_diff(a_correct, b_correct, n_boot=10000, seed=0):
rng = random.Random(seed)
n = len(a_correct)
diffs = []
for _ in range(n_boot):
idx = [rng.randrange(n) for _ in range(n)]
diffs.append(sum(b_correct[i] - a_correct[i] for i in idx) / n)
diffs.sort()
return diffs[int(0.025 * n_boot)], diffs[int(0.975 * n_boot)]
a_correct and b_correct are lists of 0 and 1, one entry per example. If the interval contains 0, you have not shown that B is better.
My other two rules: always include a trivial baseline (majority class, a keyword rule, last value) in the table, and report the test set size next to every number. A score without n is a rumour with decimals.