Skip to the content ForoHackers
Log inJoin

tired_sre_at_3am

Joined 6 January 2025

Posts in the forum

  1. Replied in Hello: freelancer, remote, and a very protected post-lunch hour

    Written runbooks for everything I do more than twice. Not for other people, for 3 a.m. me, who will not remember anything. It made on-call from home calmer, and it made my handovers short.

  2. Replied in What 'test your restores' actually means: a checklist

    Step 6 is the one that matters at 3 a.m. I have been in a room where the backup was fine, the restore tool was fine, and the key was in a password manager that lived on the server being restored. Bla…

  3. Replied in Four Windows Security event IDs I always want an alert on

    "A rule that fires 400 times a day is a rule nobody reads" should be on a poster. I would add: every alert needs an owner and a "what do I do now" line. Otherwise it is just anxiety delivered by emai…

  4. Replied in Volatile first: the order of evidence on a suspect Linux box (in your own lab)

    From the on-call side: keep a running timeline in a plain file as you go, every command with a UTC timestamp. When someone asks "what did we do and when?" two days later, nobody remembers, and this f…

  5. Replied in On-call from home: keeping the pager from eating the house

    Carl, that table is exactly the thing. Please also keep a column for "was this alert about a thing a customer would notice". If the answer is no, it is not a page.

  6. Started On-call from home: keeping the pager from eating the house

    A topic that does not get much discussion: being on-call while working from home. The blurring is real, and I have learned a few habits the slow way. **A physical boundary.** If the laptop lives in …

  7. Replied in "Tell me about a bug you caused": what a good answer looks like

    Part 3 is the one I listen for as the interviewer: do you reach for "roll back" before "find out why"? People who debug in production with the incident still burning are a different category from peo…

  8. Replied in New here: Windows admin, mostly patching

    Rings for rollouts are the quiet hero of operations. Ten machines first, wait a day, then a hundred. The incident you do not have is never in anyone's report.

  9. Replied in Trabajo remoto: escribir las cosas es la mitad del trabajo

    "Decisiones por escrito" is the thing I miss most when I switch teams. Every incident review I have attended that ended with "we already talked about this" is a missing note, not a missing memory.

  10. Replied in The skeleton of a good postmortem, and the section everybody skips

    Carl, always with the restores. You are right, and I will add it to the template. The cat on the rack from a couple of years ago remains the best example of "what was lucky": zero downtime, no idea w…

  11. Started The skeleton of a good postmortem, and the section everybody skips

    I have read a lot of postmortems and written some bad ones. The good ones share a skeleton, and it is not about blame, it is about reconstructing the story so the next person on call has a map. **1.…

  12. Replied in You cannot average percentiles (a dashboard bug I keep seeing)

    Number 3 has saved me more nights than any other alerting change. "Ratio of slow requests over a window" is a count divided by a count, it merges across hosts, and it maps directly onto what a user w…

  13. Replied in Hello from the pager side

    Agreed on both. The runbook that has only been executed at 3 a.m. by its author is a diary, not a runbook. I try to run each one once at a normal hour, with coffee, before it needs to work for real.

  14. Started Hello from the pager side

    Site reliability engineer, mostly in the habit of being awake at the wrong hours. I work on alerting, capacity and post-incident reviews, and I care a lot about blameless write-ups: the system let a …