Skip to the content ForoHackers
Log inJoin

Site reliability: start here

Site reliability
#689 · Ophuscado · · Report

Site reliability engineering applies software engineering to operating systems at scale. Instead of promising "never down", it states how reliable a service must be, measures it, and spends the gap between that target and perfection on shipping change. Reliability becomes a number you can plan with.

Key concepts

  • Service level indicators: measurements of what users feel, such as the share of requests that succeed or finish fast.
  • Service level objectives: the target for an indicator over a window, such as 99.9 percent of requests succeeding in a month.
  • Error budget: the allowed unreliability; while it lasts you ship, and when it is spent you slow down and fix.
  • Toil: repetitive manual work that grows with the service; engineers automate it away.
  • Incident response: roles, communication and calm procedures for when things break.

Vocabulary

  • SLA: a service level agreement, the promise made to a customer, often with consequences.
  • MTTR: mean time to recovery, how long restoring service takes on average.
  • Postmortem: a written, blameless account of an incident with actions to prevent a repeat.
  • Capacity planning: forecasting load so resources exist before they are needed.

What to study, in order

  1. Linux, networking and one scripting language well enough to debug a live machine.
  2. Pick a small service and define two indicators and an objective for it.
  3. Measure them, compute the error budget and chart how fast it burns.
  4. Build alerts on budget burn rather than on every blip.
  5. Run a game day: inject a failure, follow a written runbook and time the recovery.
  6. Write the postmortem, with causes, timeline and follow-up actions.

Related tags

#observability #devops #reliability #distributed-systems

Log in or join to reply.