Loading...
Loading...
Fix-sized speed versus how latency degrades as load grows 10x, and which you optimize first
Last topic you priced scale with hard numbers: 10k RPS at 100 ms hold means 1,000 concurrent slots, peak runs 5× the average, and the herd, the hot partition, and the saturated balancer punish bad guesses. Now a checkout page takes six seconds with one user at 3am, when no herd exists and every slot sits empty. Is that a performance problem or a scalability problem? Performance means how fast one request is. Scalability means how well that speed holds as load grows. The answer decides what you fix, a slow query or your whole architecture, and teams get it wrong constantly, then spend weeks fixing the wrong one.
The one-line test: slow for a single user alone at 3am? Performance problem. Fast alone but falls apart under load? Scalability problem.
How fast one request gets answered. A stopwatch on a single user.
How well that speed holds up when a crowd shows up. A stopwatch on everyone.
That is performance. Something in the request path is doing too much work: an unindexed query, meaning a database lookup with no shortcut index so it scans row by row, a 4MB hero image, a chatty API chain that makes five calls in a row. Profile it, find the slowest step, shrink it.
That is scalability. One user's request is cheap, while ten thousand at once exhaust connections, CPU, or memory. The fix is architectural: balance load, add capacity, queue the slow stuff for later.
That is both. The working rule is to fix performance first. It is usually a day of work instead of a month, and a faster request is also a cheaper one, which buys you headroom for free because it holds shared resources for less time.
You do not need forty dashboards. Five numbers tell you which problem you have. Response time percentiles, meaning the latency cutoffs like p50 for the median user and p99 for the unluckiest 1%, separate per-request work from queueing. Throughput and saturation numbers separate a heavy day from an undersized system:
| Number | What it tells you | Whose problem |
|---|---|---|
| Response time (p50, p99) | How fast requests feel, including the unlucky ones | Performance |
| Throughput (RPS) | How many requests you survive per second | Scalability |
| Error rate | Share of requests that fail outright | Both |
| CPU / memory use | How close the machines are to empty | Scalability |
| Concurrent users | How many people are leaning on it right now | Scalability |
Here is the connection people miss: a query cut from 100ms to 10ms does not just feel faster. It holds its database connection for one-tenth the time, which means the same database serves roughly ten times the traffic. Performance work is scalability work, up to a point.
Past that point, no amount of tuning saves a single server from ten thousand simultaneous users. That is the wall where architecture takes over, with more machines, balanced load, and queued background work. It is exactly why the next topics exist.
Watch out for averages. An average response of 80ms can hide 1% of users timing out at 10 seconds. Track p99, the experience of your unluckiest real users, meaning 99% are faster and 1% are slower, or you are grading yourself on a curve.
Performance and scalability meet in one equation. If each request holds a connection for 200ms, one thread handles 5 requests a second, because 1 second divided by 0.2 seconds is 5. A pool of 100 connections, which is just 100 reusable database slots shared across requests, tops out near 500 RPS (100 times 5), and then queueing starts, which users feel as a cliff, not a slope. Cut that hold time to 20ms with an index, and the same pool clears 5,000 RPS, because each slot now handles 50 requests a second. Nothing scaled. Everything scaled.
| Hold time per request | 100-connection pool clears | What changed |
|---|---|---|
| 200 ms | ~500 RPS | Unoptimized query, chatty API chain |
| 50 ms | ~2,000 RPS | Index added, payload trimmed |
| 10 ms | ~10,000 RPS | Cache hit for the hot 90% of reads |
This is Little's law in work clothes: concurrent requests equal RPS times hold time, so shrinking either factor lets the machine breathe. Past roughly 70% CPU or pool saturation, latency stops degrading gracefully and starts queueing exponentially, which is why graphs look flat and then vertical.
The classic failure looks like this: checkout is slow at noon, so the team adds servers. Checkout stays slow, because one unindexed query runs 2 seconds per order at any load. Ten servers just run the same bad query in parallel. Users see identical slowness while finance sees ten times the bill. The rejected scaling fix failed because the bottleneck was work per request, not machine count, and extra machines only multiply the same waste. The numbers expose it fast: ten boxes each clearing 5 slow requests a second still clear only 50 RPS of 2-second work, while one index cutting hold time to 20ms would have cleared 5,000 RPS on the existing pool.
The mirror image is just as common: the team tunes a query from 40ms to 12ms while the real problem is a connection pool of 20 facing 3,000 concurrent checkouts at peak. Off-peak everything sings. At 7pm the pool exhausts, threads pile up, and timeouts cascade into retries that double the load. On-call sees the retry storm, not the pool, until someone graphs queue depth and finds requests waiting hundreds deep for a free slot. The edge case that exposes it is always peak concurrency, never the quiet hour the tuning was tested in.
You now hold two numbers that refuse to be one: a 100-connection pool clearing 500 RPS at 200 ms hold time, or 5,000 RPS at 20 ms. How long one request takes and how many finish per second sound like the same thing. The next topic shows why a video call and a nightly backup can each be “fast” by one measure and broken by the other.