A distributed system is a set of computers that cooperate over a network to look like one. It exists to scale past one machine and to survive failures, and it pays for that with a new class of problems: messages are delayed or lost, machines crash halfway through, and clocks disagree.
Key concepts
- Partial failure: some parts fail while others run on, and often you cannot tell a dead node from a slow one.
- Replication: copies of data on several nodes give availability and read speed, and raise the question of keeping them in step.
- Consistency models: linearizability, sequential and eventual consistency say what a reader may see after a write.
- CAP and PACELC: during a network partition you choose between consistency and availability; even without one, between consistency and latency.
- Time and ordering: logical and vector clocks order events without trusting wall clocks.
- Partitioning: splitting data across nodes by key, and rebalancing when nodes come and go.
Vocabulary
- Idempotent: safe to apply twice, which makes retries harmless.
- Quorum: a majority whose agreement is enough to decide.
- Leader: the node that orders writes for others.
- Timeout: the only failure detector a network gives you, and an imperfect one.
- Split brain: two parts of a system each believing they are in charge.
What to study, in order
- Concurrency and networking basics: threads, sockets, what a timeout really tells you.
- The fallacies of distributed computing and the CAP result, in your own words.
- Build a replicated key-value store with a leader, and test it with killed processes and dropped messages.
- Study logical and vector clocks and implement a version vector.
- Read a consensus paper, then a paper from a real system, and write down every assumption.