A dashboard panel says "p99 latency: 300 ms". Behind it, ten servers each report their own p99, and the panel shows the average of those ten numbers. It looks reasonable. It is wrong.
A tiny example with two servers:
server A: 1000 requests, p99 = 100 ms
server B: 10 requests, p99 = 900 ms (one slow request, nine fast)
average of p99s = 500 ms
The average says 500 ms, but 99% of all 1010 requests finished in about 100 ms or less, since A carries almost all the traffic. The real combined p99 is near 100 ms, and the 500 is a number that describes nothing that happened. You can also build the opposite case where the average of the percentiles hides a real tail.
The problem: a percentile is a position in a sorted list, and positions in two different lists cannot be combined by arithmetic on the positions.
What works:
- Aggregate the raw distribution, not the summary. Keep a histogram per server with fixed bucket boundaries, add the buckets together, and compute the percentile from the merged histogram.
- If you need an approximate sketch for huge volumes, use a structure designed to be merged (a mergeable quantile sketch), and accept its documented error.
- For alerting, prefer "fraction of requests slower than X" (a ratio of two counts, which does add up) over "p99 above X".
The average of the means is fine if you weight by request counts. The average of percentiles is never fine.