Horizontal Scaling

Your server remembers too much

The last topic ended with L4 at the edge and L7 behind it routing by path prefix and hostname, both assuming any request can land on any backend. You bought three servers and that layered balancer. Traffic splits beautifully, until a user logs in on server A, clicks around, lands on server B, and gets logged out. Server B never met this person. The session lived in server A’s memory and died with the request that created it.

This is the wall every team hits going horizontal: servers with memories can’t be swapped. The fix isn’t more servers. It’s servers that forget, pushing everything worth remembering somewhere all of them can reach. A stateless server, one that keeps no per-user memory between requests, can be swapped for any other.

Two ways to grow, thirty seconds

Production norm: scale databases up, scale app servers out. Different layers, different math.

The four things that make boxes swappable

Who adds the boxes, you, or the cloud?

Manual scaling means a human watching graphs at midnight. Autoscaling means rules do it: CPU past 70% for five minutes, add two boxes. Traffic dies at 2am, shed them. Three flavors cover nearly everything:

Count the boxes before the rush counts them for you

Capacity math is one equation: concurrent requests ≈ arrival rate × average handling time. At 500 requests per second with a 200ms average response, you carry about 100 concurrent requests at any instant. If one box comfortably holds 50 concurrent connections before latency bends, you need at least two boxes, and you run three or four, because one will always be deploying, dying, or warming up.

Headroom has a price list. Running at 70% CPU leaves room for a single-box failure to redistribute without tipping the rest over; running at 90% means one dead box cascades the whole pool, because the survivors absorb 50% more load each and their latency climbs exactly when retries multiply it. The standard autoscale trigger, add capacity at 60–70% sustained CPU for 5 minutes, is not about efficiency. It is about keeping the failure of one box survivable.

The new boxes arrive after the fire is out

Autoscaling lags reality by design. Metrics aggregate over minutes, instances take 1–3 minutes to boot, and your app takes another minute to warm its caches and connection pools. A flash spike that doubles traffic in 60 seconds gets no help from autoscaling at all, the new boxes arrive to a fire already out, or already fatal. On-call sees CPU pegged at 100%, new instances “pending” in the console, and users timing out while help boots.

Cold starts make it worse. A fresh application process serves its first thousand requests 2–5× slower while its just-in-time compiler, the runtime that compiles hot code paths on the fly, warms up and its local caches fill, so the balancer’s least-connections logic, sending each request to the box with the fewest open connections, routes more traffic to the cold box because it momentarily shows the fewest connections. Scheduled scaling (up before the 9am rush), warm pools of pre-booted instances, and slow-start modes that ramp traffic to new boxes over 30–120 seconds exist precisely for this gap between “box exists” and “box is useful.”

Scaling the wrong layer leaves latency climbing

Here is the failure that surprises teams once: the autoscaler adds boxes and latency keeps climbing anyway. That happens when the bottleneck was never in the pool, if the shared database or a downstream service is saturated, new app boxes just add more mouths to a starved trough. So before trusting the instance count, check downstream queue depth and database CPU first. The working rule to carry with you is short: estimate concurrency as arrival rate times handling time, keep one spare box for failure plus headroom so that failure stays survivable, and autoscale on the user-facing signal, high percentile latency or requests per box, with a cooldown longer than a boot. Vertical scaling, a bigger single machine, buys you days of speed at the cost of a harder eventual wall; horizontal scaling, more interchangeable machines, buys a survivable wall at the cost of statelessness, discovery, and operations you must build before you need them.

The shortcut teams try first: pin users to boxes, and why it loses

Before extracting sessions into a shared store, teams try IP-hash stickiness: hash each client address so a user always lands on the same box, and let servers keep remembering. It fails on arithmetic. Uneven hashes pin 40% of users on one box while its neighbors idle, a dead box logs out exactly the users pinned to it instead of none, and every scale event re-hashes the ring and shuffles thousands of sessions at once. A shared store lookup costs roughly 1–2ms per request; a stickiness skew costs whole servers of stranded capacity. Statelessness is cheaper than the skew.

Swappable boxes still meet strangers directly

Every box you add is another liar, another clock skew, another network hop that can fail mid-sentence. Horizontal systems don’t eliminate failure, they budget for it, then route around the failed ones, sizing by arrival rate times handling time with a 60–70% CPU trigger and 30–120 second slow-start ramps so new boxes warm before they carry weight. But every one of those interchangeable boxes still shakes hands with strangers directly: real IP addresses exposed, encryption negotiated in app code, floods landing on business logic. The open question is who stands at the door so backends never have to, a gatekeeper that hides, encrypts, filters, and balances even when only one server stands behind it.