Loading...
Loading...
99.9 vs 99.99 uptime math and what breaks when replicas diverge — the intuition behind CAP
Last topic ended with two copies holding different profile photos and a hedged read at p95 papering over a 4-second straggler. Hedging fixes slowness. It says nothing about truth. You transfer $500 to a friend. The app says “done.” Your friend checks and sees nothing. Refresh, refresh, still nothing. Then it appears ten minutes later. Two servers held two different truths for ten minutes, and you saw the stale one, meaning a copy that had not yet received the latest write.
For a bank balance, that is unacceptable. For a social media like-count, nobody cares. That difference, between “must be correct right now” and “close enough for now,” is the most important trade-off in distributed systems, which simply means software running on more than one machine that must act like one system. Everything in this topic is that choice, worked through carefully.
Consistency means everyone sees the newest truth, or the system refuses to answer. No stale photos, no phantom balances. When two copies disagree, the system shows an error rather than showing a lie.
Costs you: when servers cannot reach each other, someone gets an error screen instead of an answer, because the server will not guess from an old copy.
Availability means every request gets an answer, always. It might be last minute's answer, but the app never goes dark. The server responds from whatever copy it has, even if that copy is behind.
Costs you: that answer can be outdated, and two users can see different realities for a while, until the copies catch up with each other.
The rejected hope is to always answer and always answer correctly. That works beautifully while every server can talk to every other server. It stops working the moment the network between your servers hiccups. A cable gets cut, a switch reboots, a whole datacenter goes quiet for ninety seconds. This is not hypothetical. It happens to every large system, regularly. The numbers make it concrete: a link that drops for 90 seconds a month already burns through a sixth of a four-nines budget before your code fails at all.
In that moment, Server A literally cannot ask Server B what the truth is. A network split with servers isolated on each side is called a partition. Server A faces a forced choice: stay silent until the network heals, which favors consistency, or answer from its own possibly-stale copy, which favors availability. There is no third option, because no message can cross a broken network no matter how clever the code is.
The CAP theorem, a result stating how distributed guarantees interact, says a distributed system can only guarantee two of three things. Since network failures, called partition tolerance, which means continuing to operate when the network between your own machines breaks, are a fact of life, you really only ever pick between consistency and availability:
Everyone sees the newest data
Everyone always gets an answer
Keeps working when networks split
Whoever owns the data that hurts most when it is wrong picks consistency. Whoever owns the screen the user stares at picks availability. The same company gives different answers per feature, because a wrong balance costs money while a stale feed costs almost nothing:
| System | Picks | Because |
|---|---|---|
| Bank ledger | CP | A wrong balance is worse than an error page |
| Social feed | AP | A missing post for a minute harms nobody |
| Flight seats | CP | Two people cannot own seat 14A at once |
| DNS | AP | Yesterday's address still gets you there |
Notice the pattern: money and reservations lean consistent and partition-tolerant, while eyeballs lean available and partition-tolerant. Most companies run both at once, with strict checks for payments and loose reads for everything around them. The useful question is never which one the whole system picks, but which one each feature picks.
“Eventually consistent,” which means copies converge if writes pause but says nothing about how fast, is incomplete without a number. In practice, async replication, where the primary answers first and ships copies afterward, usually lands in single-digit milliseconds to a few seconds inside one region. Across continents, expect hundreds of milliseconds at best and seconds under load. That window is the exact period two users can see two different truths, like a like-count off by three or a cart missing its last item.
| Setup | Typical staleness | Who tolerates it |
|---|---|---|
| Same-region async replicas | Milliseconds to ~1 s | Feeds, catalogs, like-counts |
| Cross-region async | Hundreds of ms to seconds | Global feeds, name-lookup style data |
| Quorum writes | Zero staleness, plus write latency | Ledgers, seat maps, inventory |
The knob behind the table is quorum size. A quorum means the minimum number of copies that must confirm before an operation counts as done, usually a majority. Suppose you keep three copies. If a write waits for two of the three to confirm, any later read that also consults two copies must overlap at least one copy that saw the write, so staleness vanishes. But every write pays the round trip to the slower of those two copies, often 1 to 5ms in one region or 100 to 200ms across an ocean. If the write returns after one copy confirms, the response is fast, and staleness equals however long the slowest remaining copy still owes you, which is exactly the replication lag above.
Walk a concrete failure with that knob set loose. Three copies hold a gift-card balance of $50. The network splits. Each side keeps accepting spends from its own copy. When the split heals, the two sides discover they each sold the same $50. No quorum overlapped both writes, so nothing stopped the double spend, and a person now has to reconcile by hand. Tighten the quorum to a majority and the minority side would have refused instead, with errors during the storm but no double spend after it. A second rejected option is routing every read to the primary to dodge staleness without quorums: it works until the primary sits across an ocean, when every read pays 100 to 200ms and the primary becomes the single point of failure you built replicas to escape.
Pick availability for money and you get the call nobody wants: two sides of a split both accept the same last seat, the same gift card balance, the same inventory unit. When the network heals, the merge logic discovers it sold what it did not have. Users see double charges or cancelled orders. Support sees a queue. Engineering writes the reconciliation script by hand, and the edge case that breaks first is always the non-idempotent action, meaning an action like charging a card that cannot safely run twice.
Pick consistency for eyeballs and you get the opposite page: a ninety-second network wobble turns the whole feed into error screens, because replicas would rather refuse than serve a minute-old post. Users see darkness over content nobody would have minded being slightly stale. The cost lands unevenly too. A consistency-first login or homepage path takes the entire site down during a split, while a consistency-first ledger tucked behind a loose storefront only pauses checkouts. That is why the strict piece usually sits deep inside, with cached and available reads in front of it.
You priced the split: same-region lag in milliseconds, cross-region in hundreds of milliseconds to seconds, and a $50 gift card spent twice when no quorum overlaps both writes. The open question is mechanical. When an available system keeps answering through that window, what machinery decides how fast the third copy catches up, and what happens to writes that landed on opposite sides?