Site reliability engineering applies software engineering to operating systems at scale. Instead of promising "never down", it states how reliable a service must be, measures it, and spends the gap between that target and perfection on shipping change. Reliability becomes a number you can plan with.
Key concepts
- Service level indicators: measurements of what users feel, such as the share of requests that succeed or finish fast.
- Service level objectives: the target for an indicator over a window, such as 99.9 percent of requests succeeding in a month.
- Error budget: the allowed unreliability; while it lasts you ship, and when it is spent you slow down and fix.
- Toil: repetitive manual work that grows with the service; engineers automate it away.
- Incident response: roles, communication and calm procedures for when things break.
Vocabulary
- SLA: a service level agreement, the promise made to a customer, often with consequences.
- MTTR: mean time to recovery, how long restoring service takes on average.
- Postmortem: a written, blameless account of an incident with actions to prevent a repeat.
- Capacity planning: forecasting load so resources exist before they are needed.
What to study, in order
- Linux, networking and one scripting language well enough to debug a live machine.
- Pick a small service and define two indicators and an objective for it.
- Measure them, compute the error budget and chart how fast it burns.
- Build alerts on budget burn rather than on every blip.
- Run a game day: inject a failure, follow a written runbook and time the recovery.
- Write the postmortem, with causes, timeline and follow-up actions.