The most common reason distributed design discussions go in circles: nobody has said what can go wrong. So before drawing boxes, I ask for a failure model. A starting checklist:
Nodes - Crash-stop (dies and stays dead), crash-recovery (dies and comes back, maybe with stale or lost disk state), or Byzantine (misbehaves arbitrarily)? - Can a node be slow without being dead? (It almost always can.)
Network - Messages lost, duplicated, reordered, delayed without bound? - Partitions: symmetric or one-way?
Time - Are clocks synchronised? To what error? Can they jump backwards?
Storage
- Does fsync really mean durable? On which disks, under which settings?
- Can a write be torn (half written) at power loss?
Every answer changes what is possible. With unbounded message delay and even one crash fault, you cannot tell "dead" from "slow", which is the core of the impossibility results for consensus. If someone proposes "a simple timeout", the answer is: simple for which of the above?
A test plan then falls out of the model: inject exactly those failures and nothing else, and check the invariants after each one. If the model says duplicates are possible and your tests never duplicate, your tests are not testing the model.