tired_sre_at_3am
Se unió el 6 enero 2025
Mensajes en el foro
-
Respondió en Hello: freelancer, remote, and a very protected post-lunch hour
Written runbooks for everything I do more than twice. Not for other people, for 3 a.m. me, who will not remember anything. It made on-call from home calmer, and it made my handovers short.
-
Respondió en What 'test your restores' actually means: a checklist
Step 6 is the one that matters at 3 a.m. I have been in a room where the backup was fine, the restore tool was fine, and the key was in a password manager that lived on the server being restored. Bla…
-
Respondió en Four Windows Security event IDs I always want an alert on
"A rule that fires 400 times a day is a rule nobody reads" should be on a poster. I would add: every alert needs an owner and a "what do I do now" line. Otherwise it is just anxiety delivered by emai…
-
Respondió en Volatile first: the order of evidence on a suspect Linux box (in your own lab)
From the on-call side: keep a running timeline in a plain file as you go, every command with a UTC timestamp. When someone asks "what did we do and when?" two days later, nobody remembers, and this f…
-
Respondió en On-call from home: keeping the pager from eating the house
Carl, that table is exactly the thing. Please also keep a column for "was this alert about a thing a customer would notice". If the answer is no, it is not a page.
-
Inició On-call from home: keeping the pager from eating the house
A topic that does not get much discussion: being on-call while working from home. The blurring is real, and I have learned a few habits the slow way. **A physical boundary.** If the laptop lives in …
-
Respondió en "Tell me about a bug you caused": what a good answer looks like
Part 3 is the one I listen for as the interviewer: do you reach for "roll back" before "find out why"? People who debug in production with the incident still burning are a different category from peo…
-
Respondió en New here: Windows admin, mostly patching
Rings for rollouts are the quiet hero of operations. Ten machines first, wait a day, then a hundred. The incident you do not have is never in anyone's report.
-
Respondió en Trabajo remoto: escribir las cosas es la mitad del trabajo
"Decisiones por escrito" is the thing I miss most when I switch teams. Every incident review I have attended that ended with "we already talked about this" is a missing note, not a missing memory.
-
Respondió en The skeleton of a good postmortem, and the section everybody skips
Carl, always with the restores. You are right, and I will add it to the template. The cat on the rack from a couple of years ago remains the best example of "what was lucky": zero downtime, no idea w…
-
Inició The skeleton of a good postmortem, and the section everybody skips
I have read a lot of postmortems and written some bad ones. The good ones share a skeleton, and it is not about blame, it is about reconstructing the story so the next person on call has a map. **1.…
-
Respondió en You cannot average percentiles (a dashboard bug I keep seeing)
Number 3 has saved me more nights than any other alerting change. "Ratio of slow requests over a window" is a count divided by a count, it merges across hosts, and it maps directly onto what a user w…
-
Respondió en Hello from the pager side
Agreed on both. The runbook that has only been executed at 3 a.m. by its author is a diary, not a runbook. I try to run each one once at a normal hour, with coffee, before it needs to work for real.
-
Inició Hello from the pager side
Site reliability engineer, mostly in the habit of being awake at the wrong hours. I work on alerting, capacity and post-incident reviews, and I care a lot about blameless write-ups: the system let a …