Loading...
Loading...
Fail-over, replication, and availability numbers
Last topic ended at 2am with three jobs: notice the death, decide the next leader without crowning two, and take over without resurrecting the old one. Before building that machinery, price the goal. A vendor promises “three nines,” which is 99.9% uptime, meaning the service answers 999 out of every 1,000 minutes. Sounds excellent. It means eight hours dark every year. Four nines, meaning 99.99%, allows fifty-two minutes. Five nines allows five minutes, with roughly ten times the engineering bill of four. Every extra nine is a zero added to the budget, so the first question is always: how dark may we go? The arithmetic is just 8,760 hours a year times the failure share: 0.1% of a year is 8.77 hours, 0.01% is 52.6 minutes, 0.001% is 5.26 minutes.
| Promise | Dark per year | Dark per month |
|---|---|---|
| 99% (two 9s) | 3.65 days | 7.31 hours |
| 99.9% (three 9s) | 8.77 hours | 43.83 min |
| 99.99% (four 9s) | 52.60 min | 4.38 min |
| 99.999% (five 9s) | 5.26 min | 26.30 sec |
Most products live happily at three or four nines. Five is for pacemakers and phone switches, priced accordingly, because the testing, redundancy, and staffing behind those last minutes cost more than everything before them. A second rejected option is buying five nines by policy: unless lost minutes cost more than the tenfold engineering bill, the fourth nine is where sane teams stop.
A server will die. The rejected hope is that it will not, or that someone will be awake to rebuild it. Production replaces that hope with a posture decided ahead of time: a cold standby that wakes on failure, or two workers sharing the load so a death changes nothing:
Active-passive means one server serves while a second waits. The waiter watches heartbeats, meaning small “I am alive” pings every second or two, and when enough go missing it takes over, serving in minutes. Simple and cheap. But the standby earns nothing while waiting, takeovers take minutes while data and connections move, and anything the old primary had not yet copied over dies with it.
Active-active means both servers serve traffic at once, so a death just shifts load to the survivor. Failover is instant because there is nothing to “fail over to.” The survivor was already working. The price is coordination: two writers need conflict rules for who wins when both change the same record, and everything costs roughly double to build and run.
Chains multiply failure while spares divide it. A request crossing three 99.9% services in a row sees roughly 99.7%, since 0.999 times 0.999 times 0.999 is 0.997, because each link donates downtime. But two 99% boxes where either suffices? Failure needs both down at once: 1 minus 0.01 times 0.01 equals 99.99%. That single insight, that series hurts and parallel heals, prices every redundancy decision you will ever make. Walk it once with money attached: three chained dependencies at three nines each burn through your whole four-nines budget of 52 minutes a year before your own code fails at all.
Uptime = Uptime(A) × Uptime(B)99.9% × 99.9% ≈ 99.8%. Every dependency spends some of your nines, so each synchronous call in the request path is a direct withdrawal from the budget.
Uptime = 1 − (1 − A) × (1 − B)Two 99% boxes give 99.99%. Redundancy manufactures nines out of mediocrity, but only if failover is automatic and tested, because a standby nobody ever drills is a hope, not a nine.
Replication, meaning keeping extra copies of the same data on other machines, forces a wait-or-risk choice on every write. The rejected version answers instantly and assumes copies will catch up. That is fast until a crash eats the last seconds of truth. The three real postures differ only in how long the write waits:
Synchronous replication means the write returns only after every copy confirms. Crash a second later and nothing is lost, but every write waits on the slowest copy, and one sick replica freezes all writes. A single stalled disk at 500ms turns every write into a 500ms write, which is when teams learn that correctness has a latency price on sunny days too.
Asynchronous replication means the write returns instantly while copies catch up behind. Users feel speed, and inherit a window, often milliseconds to seconds, where a crash eats the last seconds of truth. If the primary dies with 2 seconds of unshipped writes at 5,000 writes a second, roughly 10,000 writes are gone. That multiplication is the real cost of “fast.”
Semi-synchronous replication means waiting for one other copy, not all. Most of the safety, a fraction of the wait. It survives one death, because two machines hold every acknowledged write, while a write pays one round trip instead of waiting on the slowest replica in the fleet. That is why it is the pragmatic default for ledgers that cannot afford full-sync stalls or async holes.
Failover has three jobs: notice, decide, and take over, and each has a knob with a number on it. Heartbeats notice: replicas ping the primary every 1–5 seconds, and after 3 missed beats, roughly 10–15 seconds, the primary is declared dead. That threshold is the whole trade: shorter means faster failover and more false funerals every time the network hiccups for a second. Measure it against the nines table, because a 15-second detection plus a 2-minute takeover burns a quarter of a four-nines yearly budget in one incident.
If every standby promotes itself, a network split crowns two primaries that both accept writes, which is split-brain, meaning two leaders diverging, and the merge afterward is manual. Production systems elect with a quorum, meaning only the side holding a majority of votes may lead, using coordination protocols such as Raft or Paxos, which are algorithms for getting a majority to agree on one leader, or a coordination service in that style. The minority goes silent instead of going rogue. That silence is consistency-first behavior doing its job: refusing is cheaper than diverging.
The old primary may not know it is dead. Fencing, meaning actively blocking the old leader from writing again, keeps it buried, by revoking its lease, which is a time-limited leadership ticket, by bumping a generation number, also called an epoch, that storage checks on every write so old generations get rejected, or by cutting its network access. Database failover managers, streaming-controller elections, and cloud load-balancer health checks, typically checking every 5 seconds and needing 2 healthy checks to add a backend and 3 unhealthy to remove one, all encode the same idea: the new leader's first act is making sure the old one stays buried.
Answer availability with two numbers, not one. RTO, meaning recovery time objective or how long until service returns, runs minutes for standby setups and seconds where both sides already serve. RPO, meaning recovery point objective or how much data you may lose, is zero with synchronous replication and seconds with async. A system with a 10-minute RTO cannot promise four nines no matter what the replicas look like, because one real incident eats the 52-minute yearly budget in a sixth of the allowance, and two incidents break it outright.
The same math disciplines the active-active question. Two writers are only strictly better where conflicts are cheap, like feeds where a merged order harms nobody. Where both sides can spend the same dollar or book the same seat, the merge logic often costs more than the downtime it prevents, which is why ledgers stay standby-based while feeds run both sides hot. Manual runbooks do not count toward nines either, because a human who must wake up and type commands is a 15-minute RTO wearing optimism.
You priced uptime: 10 to 15 seconds to notice, RTO in minutes for standby or seconds where both sides serve, RPO at zero with sync or seconds with async, and fencing through leases and epochs so the old leader stays buried. That machinery assumes small “I am alive” pings and election votes actually cross the network. What carries them, hop by hop, from one machine to another?