Loading...
Loading...
Round robin, least connections, IP hash, health checks and how active-passive differs from active-active
The previous topic closed with static content solved: at a 98% edge hit ratio only 200 of 10,000 requests per second ever reach origin, and the leftover question was which server takes the next dynamic request the edge cannot answer. You did the right thing for that half: three identical servers instead of one. Then server #2 runs out of memory on a Tuesday night. Users whose requests land there get error pages. Users on #1 and #3 are fine. Nobody told the traffic to stop going to the dead box, because nothing is watching.
You need something in front that sees all three, knows which are alive, and hands each incoming request to a healthy one. That distributor is a load balancer, a dedicated machine or service whose whole job is spreading requests across healthy backends, and it’s the first thing you install the day one server stops being enough.
Restaurant version: a host who seats guests across sections so no waiter drowns while another polishes glasses. Same job, packets instead of people.
Load spreads instead of piling onto whichever box answered first.
Health checks pull dead servers out of rotation in seconds. Users never notice.
Add a fourth box mid-rush or retire one for maintenance. Traffic just flows around.
Users see one address. Real server IPs, TLS termination, the decryption of browser encryption at the edge so backends handle plain traffic, and filtering all live behind it.
“Send it somewhere healthy” has more than one correct answer. The strategy you pick is a bet about what your traffic looks like:
Request 1 → A, request 2 → B, request 3 → C, repeat. Zero thinking required.
reaches for it when every server is identical and every request costs about the same.
Same rotation, but a server twice as beefy gets twice the turns. Say A has weight 3 and B has weight 2: out of every five requests, A takes three.
reaches for it when your fleet is a mix of old and new hardware.
Every request goes to whoever is currently juggling the fewest. When one request takes 50ms and the next takes 5 seconds, turn-taking breaks, this doesn’t.
reaches for it when request costs vary wildly, like video calls or file uploads.
Hash the client’s address to pick a server, so a user keeps landing in the same place. Useful when servers remember things, though making servers stateless is usually the better fix.
reaches for it when legacy sessions live on specific boxes and you can’t move them yet.
Pick blindly. Sounds lazy; with hundreds of identical servers the math evens out beautifully, and there’s nothing to misconfigure.
reaches for it when the fleet is huge, uniform, and stateless.
A strategy means nothing if the balancer keeps sending users to a dead server. So it watches, two ways. Passive watching notices real user requests failing. Active watching pings a /health endpoint every few seconds whether or not users exist. Either way, failed servers leave the rotation until they prove they’re alive again.
| Balancer | What it is | Grab it when… |
|---|---|---|
| NGINX | Software web server that also proxies and balances HTTP | You want a reverse proxy too |
| HAProxy | Software balancer built for raw TCP and HTTP speed | TCP and HTTP at serious volume |
| AWS ALB / ELB | Managed balancers from Amazon's cloud, no servers to feed | You live on AWS and hate toil |
| Envoy | Modern proxy designed for microservice meshes | Microservices with sidecars |
Say your balancer pings /health every 10 seconds and needs 3 straight failures to evict. A server dies at second 0. The balancer notices at second 30, and for those 30 seconds, one third of your users (with three servers) eat errors. Shorten the interval and you notice faster, but every backend now spends its life answering health pings instead of users.
| Knob | Sane starting value | What happens if you push it |
|---|---|---|
| Check interval | 5–10s (active HTTP check) | 1s finds death fast but hammers backends; 30s lets dead servers serve errors for half a minute |
| Unhealthy threshold | 2–3 consecutive failures | 1 failure evicts on any blip (flapping); 5 failures keeps a dead box rotating for nearly a minute |
| Healthy threshold | 2 consecutive successes before rejoin | Rejoin on 1 success and a half-booted server gets crushed instantly |
| Check timeout | 2–5s per probe | Longer than the interval means probes pile up; shorter than the p99, the latency only 1% of requests exceed, means healthy-but-slow servers look dead |
Back-of-envelope detection math: time-to-evict ≈ interval × unhealthy threshold. At 10s × 3, that is 30 seconds of a dead server receiving its full share. With N servers and round robin, roughly 1/N of requests fail during that window, on a 3-box fleet at 900 requests per second, that is around 9,000 failed requests before the corpse leaves rotation. This is why teams land near 5s × 2: about 10 seconds of pain instead of 30, at the cost of a cheap GET every 5 seconds per server.
Before buying a balancer, most teams try DNS round robin, which means publishing three addresses under one name and letting clients rotate through them. It fails in three measurable ways. DNS answers cache for minutes, so with a 300-second TTL a dead box keeps receiving roughly a third of traffic for five full minutes after death, against about 10 seconds with 5-second active checks and a threshold of 2. DNS carries no health signal, so nothing ever evicts the corpse. And clients control the rotation, not you, so uneven client caches skew the split 60/40 with no warning. DNS answers where something was. A balancer answers what is healthy right now, and that freshness is the whole purchase.
Balancers fail in ways that look nothing like a dead backend. The classic one is connection exhaustion: each balancer holds one connection from the user and one to the backend, so 50,000 concurrent users means 100,000 open network connections on one box. Hit the file-descriptor or ephemeral-port ceiling, the operating system limits on open files and outgoing ports, and new users get connection refused while every backend sits idle at 20% CPU. On-call sees “all servers healthy, site down,” which is the signature of a saturated middle, not a broken edge.
The second classic is the slow-backend pileup under least-connections routing. One server gets sluggish (a stuck disk, a bad deploy) and holds its connections open ten times longer. The balancer, trying to be fair, sends it fewer new requests, correct, but the requests already stuck there hold user connections open until timeouts fire, and retries from impatient clients double the load on the healthy boxes. Uneven hashing does the same thing deterministically: one overweight shard gets 40% of traffic while the dashboard average looks fine.
Start with round robin over stateless backends, servers holding no per-user memory between requests, behind active health checks at a 5-second interval, and fix stickiness by moving sessions into a shared store rather than hashing users to boxes. Every smarter strategy buys better worst-case behavior at the cost of balancer state, least-connections needs live connection counts, IP hash needs a stable ring, so only pay for cleverness once uniform rotation measurably fails. That is also why almost everyone starts with round robin even though least-connections sounds strictly smarter: least-connections only wins when request costs vary a lot, and it costs the balancer per-request bookkeeping plus failure modes of its own, a slowly dying server accumulates long-lived connections the algorithm can see but cannot fix, while round robin stays dumb, stateless, and predictable. Start dumb, measure the spread in request durations, and upgrade when the ratio of worst-case to typical latency earns it. Measure first.
Notice the limit of everything above: round robin over stateless backends, servers holding no per-user memory between requests, behind 5-second active checks routes without reading a single byte inside the request, and balancers run in pairs because the distributor itself is the last single point of failure. That blindness is fast, but it cannot send video uploads to the big machines, beta users to the beta build, or Europeans to Europe, because it cannot tell an image from a login. The open question is whether the balancer should stay blind and fast or learn to read, and what each byte of understanding costs in latency and CPU.