Saltar al contenido ForoHackers

Cardinality: the metric label that ate the monitoring server

DevOpsObservabilidadFiabilidad de sitios (SRE)
#1047 · cron_job_carl · · Denunciar

A story, blameless as usual, because the person who did it was me.

We added a label to a request counter to debug one customer's problem:

http_requests_total{route="/api/orders", status="200", user_id="48213"}

It worked. It was useful. Three weeks later the monitoring server was out of memory.

A time-series database stores one series per unique combination of label values. route has 40 values, status has 6, user_id has 200,000. Worst case: 40 x 6 x 200,000 = 48 million series, each with its own memory and index entries. The cost of a metric is not the number of samples, it is the number of series.

Rules I follow now:

  • Labels must have bounded values you can list in advance: route templates (/api/orders/:id, not the actual id), status class, region.
  • Anything unbounded (user id, session id, request id, full URL, email) belongs in logs or traces, which are designed for high-cardinality data.
  • Alert on series count growth per metric. A sudden jump is a bug report written in numbers.

Boring metrics, boring graphs, boring weekend.

#1556 · git_blame_myself · · Denunciar

We did it with a pod_name label on a short-lived job. Every run created new pods, so the series count grew forever, and old series lingered in memory. The fix was dropping the label at ingest. I also removed my name from the commit.

#1557 · c0ld_b00t · · Denunciar

From the forensics side: the same user id in a log line is gold during an investigation, and a horror inside a metric. Right tool for the right kind of data. Just keep personal identifiers out of both if you can, and out of long-term storage always.

#1558 · yak_shaving_daily · · Denunciar

I noticed this in my homelab with a per-device label for every phone that touched the wifi. The Raspberry Pi running the monitoring stack did not enjoy it. Lesson: homelab is production if it is the thing you check when something is down.

#1559 · race_condition_rita · · Denunciar

Cheap safeguard: set a hard limit on series per target and drop the whole scrape (loudly) when it is exceeded. A broken metric is better than a dead monitor. State the failure mode in advance and choose it.

Inicia sesión o únete para responder.