Saltar al contenido ForoHackers

Alert on burn rate, not on every blip

ObservabilidadFiabilidad de sitios (SRE)
#1017 · cron_job_carl · · Denunciar

If your pager goes off for every spike in errors, you will start ignoring it. A cleaner approach is to alert on how fast you are spending your error budget.

Setup: an SLO of 99.9% success over 30 days. The budget is 0.1% of requests, which is about 43 minutes of total failure in a 30-day window (30 days = 43,200 minutes, times 0.001).

Burn rate = observed error ratio / allowed error ratio. At burn rate 1 you use exactly the whole budget in 30 days. At burn rate 14.4 you use 2% of the monthly budget in one hour (14.4 x 1h / 720h = 2%).

A pattern that works for me:

Alert Burn rate Long window Short window Action
Page 14.4 1 hour 5 min wake a human
Page 6 6 hours 30 min wake a human
Ticket 1 3 days 6 hours look on Monday

The short window makes the alert reset quickly once the problem is fixed. The long window keeps one noisy minute from paging anyone. Both must be true.

Boring, calm, predictable. Pages that mean something. Also: write down what the human is supposed to do when the page fires, or you just built an expensive doorbell.

#1449 · git_blame_myself · · Denunciar

We had the opposite: 40 alerts for one dead database, because every service that talked to it screamed separately. Alerting on symptoms users can see (error ratio at the edge) instead of causes (CPU, queue depth) cut our pages to about a fifth. Confession: I wrote half of the old alerts, so that was a humbling week.

#1450 · race_condition_rita · · Denunciar

One caveat: burn rate over a ratio is blind to very low traffic. Two requests, one failure, 50% errors, a page at 4 a.m. Add a minimum request count to the expression or accept a synthetic probe for quiet services.

#1451 · cron_job_carl · · Denunciar

Yes. A minimum-volume guard is in every rule I keep. And a probe that runs every minute is boring, which is the goal.

Inicia sesión o únete para responder.