Skip to the content ForoHackers
Log inJoin

Before you design the protocol: write down the failure model

Distributed systemsReliability engineeringTesting
#988 · race_condition_rita · · Report

The most common reason distributed design discussions go in circles: nobody has said what can go wrong. So before drawing boxes, I ask for a failure model. A starting checklist:

Nodes - Crash-stop (dies and stays dead), crash-recovery (dies and comes back, maybe with stale or lost disk state), or Byzantine (misbehaves arbitrarily)? - Can a node be slow without being dead? (It almost always can.)

Network - Messages lost, duplicated, reordered, delayed without bound? - Partitions: symmetric or one-way?

Time - Are clocks synchronised? To what error? Can they jump backwards?

Storage - Does fsync really mean durable? On which disks, under which settings? - Can a write be torn (half written) at power loss?

Every answer changes what is possible. With unbounded message delay and even one crash fault, you cannot tell "dead" from "slow", which is the core of the impossibility results for consensus. If someone proposes "a simple timeout", the answer is: simple for which of the above?

A test plan then falls out of the model: inject exactly those failures and nothing else, and check the invariants after each one. If the model says duplicates are possible and your tests never duplicate, your tests are not testing the model.

#1344 · mutex_mike · · Report

Add one under network: "messages can arrive after the sender has crashed". People assume a message from a dead node cannot appear. It can, and it will be processed as if the node were alive.

#1345 · race_condition_rita · · Report

Yes. Stale messages from a dead leader are behind half of the bugs I debug. That is where epoch numbers or fencing tokens come in.

#1346 · legacy_cobol_whisperer · · Report

On mainframes we wrote the failure model into the operating procedure: "if the machine loses power between these two steps, here is the recovery". It took the form of a runbook rather than a theorem, but the thinking was the same.

#1347 · packet_loss_poet · · Report

A partition is a story where both sides think they are right. Write the failure model.

Log in or join to reply.