Skip to the content ForoHackers
Log inJoin

Reliability engineering: start here

Reliability engineering
#3219 · · Quote · Report

Reliability engineering is the discipline of making systems keep working for as long as they are needed, and of knowing how likely they are to. It applies to hardware, software and the two together. Everything fails eventually, so the work is to measure how, find the weak points and design so that failures are rare, contained and survivable.

Key concepts

  • Failure rate and the bathtub curve: early failures, a long stable period, then wear-out.
  • Redundancy: spare components, replicated services and voting, so one fault does not stop the system.
  • Availability and its arithmetic: uptime depends on how often things fail and how fast they are repaired.
  • Fault tolerance: detecting, isolating and recovering from faults while still serving.
  • Reliability testing: stress, soak and accelerated tests that provoke failures early.

Vocabulary

  • MTBF and MTTR: mean time between failures, and mean time to repair.
  • Single point of failure: one part whose loss stops the whole system.
  • Graceful degradation: losing some function but not everything.
  • Burn-in: running parts under stress to remove early failures before delivery.

What to study, in order

  1. Probability basics: exponential and Weibull distributions, and what a failure rate means.
  2. Compute availability for a service made of components in series and in parallel.
  3. Draw a reliability block diagram of a system you know and find its single points of failure.
  4. Inject failures, such as killing processes and dropping links, and watch the system recover.
  5. Collect failure data from a real system and fit it to a distribution.

Related tags

#fault-analysis #sre #distributed-systems #probability

You are not logged in, so this goes up anonymously: no account, no name and no address is stored with it, and it may carry no link and no image. Log in or join to post under your name. Our web server's logs keep the address and time of every request for up to 14 days; to keep yours out of them, use Tor. To reach the staff in confidence, use the drop. Sign as a returning source (a key kept only in this browser)

Sending takes a few seconds here: your browser first works out a proof of work against spam.

Reply