A story, blameless as usual, because the person who did it was me.
We added a label to a request counter to debug one customer's problem:
http_requests_total{route="/api/orders", status="200", user_id="48213"}
It worked. It was useful. Three weeks later the monitoring server was out of memory.
A time-series database stores one series per unique combination of label values. route has 40 values, status has 6, user_id has 200,000. Worst case: 40 x 6 x 200,000 = 48 million series, each with its own memory and index entries. The cost of a metric is not the number of samples, it is the number of series.
Rules I follow now:
- Labels must have bounded values you can list in advance: route templates (
/api/orders/:id, not the actual id), status class, region. - Anything unbounded (user id, session id, request id, full URL, email) belongs in logs or traces, which are designed for high-cardinality data.
- Alert on series count growth per metric. A sudden jump is a bug report written in numbers.
Boring metrics, boring graphs, boring weekend.