Skip to the content ForoHackers
Log inJoin

You cannot average percentiles (a dashboard bug I keep seeing)

Data scienceObservabilitySite reliability
#1093 · cron_job_carl · · Report

A dashboard panel says "p99 latency: 300 ms". Behind it, ten servers each report their own p99, and the panel shows the average of those ten numbers. It looks reasonable. It is wrong.

A tiny example with two servers:

server A: 1000 requests, p99 = 100 ms
server B:   10 requests, p99 = 900 ms   (one slow request, nine fast)
average of p99s = 500 ms

The average says 500 ms, but 99% of all 1010 requests finished in about 100 ms or less, since A carries almost all the traffic. The real combined p99 is near 100 ms, and the 500 is a number that describes nothing that happened. You can also build the opposite case where the average of the percentiles hides a real tail.

The problem: a percentile is a position in a sorted list, and positions in two different lists cannot be combined by arithmetic on the positions.

What works:

  1. Aggregate the raw distribution, not the summary. Keep a histogram per server with fixed bucket boundaries, add the buckets together, and compute the percentile from the merged histogram.
  2. If you need an approximate sketch for huge volumes, use a structure designed to be merged (a mergeable quantile sketch), and accept its documented error.
  3. For alerting, prefer "fraction of requests slower than X" (a ratio of two counts, which does add up) over "p99 above X".

The average of the means is fine if you weight by request counts. The average of percentiles is never fine.

#1724 · tired_sre_at_3am · · Report

Number 3 has saved me more nights than any other alerting change. "Ratio of slow requests over a window" is a count divided by a count, it merges across hosts, and it maps directly onto what a user would call "slow".

#1725 · overfit_otter · · Report

The statistical version of the same warning: any summary that discards the sample size cannot be combined correctly. A percentile from 10 requests and one from 1000 requests are not equally trustworthy, and the average treats them as equal.

#1726 · mutex_mike · · Report

Related trap: measuring latency at the server only sees the requests the server accepted. A request that waited in a queue before being accepted is not in your histogram. Under overload your p99 looks great because the slowest requests never made it into the data. (People call this coordinated omission.)

#1727 · selenium_sleepwalker · · Report

Same thing in front-end performance testing: "median page load of the 100 runs" is easy, but when I take the median of medians over several test machines I get a number nobody can reproduce. Keep the raw timings and compute once.

#1728 · cron_job_carl · · Report

Yes. Store raw or store histograms, never store the percentile and try to combine it later. Boring, and correct.

#1729 · bayes_and_bagels · · Report

For anyone who wants to see this in a few lines of R: simulate two samples of different sizes and compare quantile(c(a, b), 0.99) against the average of the two quantile(a, 0.99) and quantile(b, 0.99). The gap is never zero and often huge. Seeing it once is worth ten explanations.

Log in or join to reply.