Skip to the content ForoHackers
Log inJoin

The skeleton of a good postmortem, and the section everybody skips

Incident responseSite reliabilityTechnical writing
#1122 · tired_sre_at_3am · · Report

I have read a lot of postmortems and written some bad ones. The good ones share a skeleton, and it is not about blame, it is about reconstructing the story so the next person on call has a map.

1. Summary in three sentences. What broke, who was affected, how long. If someone reads only this, they should not be misled.

2. Timeline, in UTC, with sources. Not "around 3", but 03:12 alert fired (pager log), 03:19 first responder acked, 03:41 rollback started. Mark what is known (from logs) and what is remembered (from people). Memory is the weakest data source in the room.

3. What went well. Yes, really. The alert that fired on time, the runbook that was correct. These are things to protect.

4. What was lucky. The part I find most valuable: what could have made this much worse and didn't? The backup that happened to be fresh? The person who happened to be awake? Luck is a risk you have not mitigated yet.

5. Contributing factors, not "root cause". Complex failures rarely have one. Ask "why was this possible?" instead of "who did this?"

6. Action items with an owner and a date. An action item with no owner is a wish. Keep them few. Ten actions means none gets done.

The test of a good postmortem: could a new engineer read it six months later and know what to do differently on the next alert?

#1821 · git_blame_myself · · Report

"What was lucky" is the section I steal for everything. In my last postmortem the luck was that the broken config was pushed on a Friday morning, not at 17:55. I put that in writing and the team finally agreed on a no-late-Friday-merges rule without anybody being the villain.

#1822 · cron_job_carl · · Report

Add a line to the timeline for "backups verified, last restore test passed" if there is data loss risk. It answers the first question management asks: "can we get it back?" And as always, if the answer is "we think so", you know what you are doing next week.

#1823 · tired_sre_at_3am · · Report

Carl, always with the restores. You are right, and I will add it to the template. The cat on the rack from a couple of years ago remains the best example of "what was lucky": zero downtime, no idea why.

#1824 · rubber_duck_supervisor · · Report

Question back at the thread: in your last postmortem, which section did people argue about the most? I would bet on "contributing factors", because it is where "who" sneaks back in disguised as "what".

Log in or join to reply.