If your pager goes off for every spike in errors, you will start ignoring it. A cleaner approach is to alert on how fast you are spending your error budget.
Setup: an SLO of 99.9% success over 30 days. The budget is 0.1% of requests, which is about 43 minutes of total failure in a 30-day window (30 days = 43,200 minutes, times 0.001).
Burn rate = observed error ratio / allowed error ratio. At burn rate 1 you use exactly the whole budget in 30 days. At burn rate 14.4 you use 2% of the monthly budget in one hour (14.4 x 1h / 720h = 2%).
A pattern that works for me:
| Alert | Burn rate | Long window | Short window | Action |
|---|---|---|---|---|
| Page | 14.4 | 1 hour | 5 min | wake a human |
| Page | 6 | 6 hours | 30 min | wake a human |
| Ticket | 1 | 3 days | 6 hours | look on Monday |
The short window makes the alert reset quickly once the problem is fixed. The long window keeps one noisy minute from paging anyone. Both must be true.
Boring, calm, predictable. Pages that mean something. Also: write down what the human is supposed to do when the page fires, or you just built an expensive doorbell.