Failure Models and Replication
A distributed system has to say what kinds of failure it tolerates: a machine that stops, a message that never arrives, a machine that stops and comes back with old state. Replication is the usual answer, and it turns one copy's truth into a question about which copy to believe.
What a learner can do afterwards
- Explain why a crashed node and a slow one look the same from outside, and why that is hard to resolve
- Explain what a primary and its replicas each promise a client
- Say what a replica that rejoins with stale state has to do before serving reads
1 · Read
You already know messages carry order between machines. Now ask what can go wrong. A machine can stop dead, a message can vanish, or a stopped machine can return later still believing old state. First you must name which failure you tolerate.
You call a node and hear nothing. It may have crashed, or it may just be slow. From outside both look identical: silence. Systems set a timeout so they can act after a wait, but a timeout still cannot tell the two apart for sure.
Replication is the usual answer: keep copies on several machines. The primary takes writes and promises one order, while replicas copy it and promise to serve only what they have caught up with. One copy's truth becomes a question about which copy to believe.
A replica that rejoins with stale state must catch up before serving reads. It copies the changes it missed first, and only then answers clients. Otherwise it hands out yesterday's truth.
Name the failures you tolerate, remember silence hides crash and slowness alike, and never let a stale replica answer.
Original instructional content authored for os-taxonomy.
2 · Watch
Take it off screen
Where it sits
Learn first
Optional connections
8 questions wait behind this lesson, each with its answer explained. Every answer feeds the sky: stars light as they are learned, and dim when it is time to come back.