Loading...
Loading...
p50/p99 latency versus requests per second, Little's Law, queuing and why optimizing one can hurt the other
Last topic you turned 200 ms of hold time into 500 RPS on a 100-connection pool, then into 5,000 RPS at 20 ms. That math counts completions per second. It says nothing about how one video frame feels. A video call that lags half a second is unusable, while a nightly backup that takes six hours is completely fine. Same industry, opposite needs, because “fast” splits into two different questions: how long does one thing take, and how many things finish per second?
Mix them up and you optimize the wrong one: a backup that answers instantly but processes one file an hour, or a call that moves mountains of data with a two-second delay. Both are “fast” by one measure and broken by the other. The fix starts by naming which clock you are trying to beat. Latency, meaning the delay for a single request from send to answer, is the first clock. Throughput, meaning completions per second across the whole system, is the second.
How long one thing takes
Latency is the delay for a single request, measured from the moment it leaves until the answer lands. Start the clock when a request leaves. Stop it when the answer arrives. That duration, milliseconds usually, is latency, and it is what an individual user feels.
Feel it: tapping a link and counting “one Mississippi” before anything happens. That wait is latency you can feel in your bones.
How many finish per second
Throughput is the completion rate of the whole system, usually counted as requests per second, abbreviated RPS. Count completed requests over a second and that rate is throughput. It measures the system's capacity, not any single user's wait.
Feel it: ten thousand people clearing the gates per minute. Nobody cares how fast one person walks out. What matters is the rate the whole crowd drains.
Think of water in a pipe. Latency is how long the first drop takes to travel end to end. Throughput is how many gallons arrive per minute. A thin pipe delivers its first drop quickly but barely trickles. A wide pipe moves lakes of water, yet its first drop still takes the full trip to arrive.
Networks behave like this. Fiber across an ocean has unavoidable latency, because light itself needs roughly 150ms for a round trip to another continent, while carrying very large throughput. No engineering fixes the first number, because it is set by distance. The second number is where capacity planning and money go.
Memorize the shape of this table, not the digits: memory is roughly a million times faster than a cross-country network hop. Every time your code crosses that gap casually, you pay for it. The table lists typical latencies, meaning the time one operation takes, from fastest to slowest:
| Operation | Time |
|---|---|
| L1 cache reference, meaning the CPU's closest tiny memory | 0.5 ns |
| Main memory reference | 100 ns |
| SSD random read | 150 μs |
| Round trip within same datacenter | 500 μs |
| Disk seek | 10 ms |
| Send packet across the ocean and back | 150 ms |
Source: Jeff Dean, Google. A request that touches disk and crosses an ocean is doing the two slowest things on this list back to back. That is why one careless remote call can dominate an entire response budget.
The rejected fix is to demand both at once: batch everything for efficiency and still answer instantly. Batching, which means collecting many items and processing them together, breaks that hope in a specific way. Want more throughput? Batch work together, but each item now waits for the batch to fill, so latency climbs. Want less latency? Handle everything instantly with no batching, and watch total capacity drop because per-item overhead returns. The measured cost is concrete: batching 1,000 rows can lift a pipeline from 200 to 5,000 rows a second while the first row waits seconds for the batch to fill. There is no setting where both numbers are perfect. There is only the balance your users actually need.
Gaming, trading, video calls, autocomplete. If a human is staring at a spinner or money moves on the answer, latency is the whole game. Here a 100ms budget for the full interaction is the usual line between feeling instant and feeling broken.
Nightly ETL, meaning extract-transform-load jobs that move bulk data between systems, plus log pipelines, video encoding, and bulk email. Nobody watches these run. What matters is clearing the mountain by morning, so large batches and deep queues are correct here.
Users feel latency in thresholds, not averages. Around 100ms an interaction feels instant. Past a second without feedback, attention breaks and people retry, which adds load on top of load. Throughput, meanwhile, is provisioning math. Say your API averages 2 KB responses at 10k RPS: that is 2,000 bytes times 10,000, which is 20 MB/s, or 160 Mbps sustained once you multiply bytes by 8 to get bits, before retries and encryption overhead. Double it for headroom and you know what network card and how many boxes you actually need.
| Budget | Number to memorize | Why it bites |
|---|---|---|
| Feels instant | ~100 ms end to end | Autocomplete, taps, hover states live or die here |
| Attention breaks | ~1 s without feedback | Past this, show progress or absorb a retry storm |
| Cross-continent floor | ~150 ms round trip minimum | Physics, not code. Only closeness fixes it |
| Throughput headroom | Provision 2× average, shed past it | Queues explode near 100%. Never run hot |
Working habit worth building: state the latency budget first, for example p99 under 200ms, where p99 means the latency 99% of requests beat and 1% exceed. Then size throughput for peak plus retries. Systems that meet average RPS but miss p99 still read as broken, because the unluckiest users hit every timeout.
The failure mode is always the same shape: someone optimizes the number they can graph and breaks the one users feel. Batch writes of 1,000 rows lift a pipeline from 200 to 5,000 rows a second, and the first row now waits seconds for the batch to fill. Great for the nightly job, fatal for the live checkout sharing the same queue.
Message consumers tuned to wait 30 seconds per batch, network buffering that holds small sends to fill larger packets, or a database layer flushing 500 inserts at once: throughput graphs climb while a single user's save hangs. On-call sees healthy throughput with angry users. When this breaks first, the p99 for interactive saves jumps into seconds while the average looks fine. The fix is separate lanes, with small fast batches for interactive traffic and big slow ones for bulk, so the bulk lane can never starve the human lane.
Garbage-collection pauses, which are runtime freezes to reclaim memory, noisy neighbors on shared disks, or one slow replica in a fan-out, where one request fans out to many backends and waits for all of them: 99 requests finish in 40ms, one takes 4 seconds, and the page waits for the slowest. Users report “sometimes it hangs.” When this breaks, adding capacity does nothing because the problem is variance, not volume. The fix is hedging, which means sending the same read twice after the 95th percentile delay, where p95 means 95% of requests have already finished, and taking whichever answers first. A second rejected option is overprovisioning to fix variance: doubling boxes from 20 to 40 leaves the 4-second straggler untouched, so p99 stays pinned while the bill doubles.
You priced both clocks: 100 ms feels instant, 150 ms is the ocean floor no code crosses, and hedging at p95 rescues the 4-second straggler that capacity cannot fix. But two servers holding the same profile photo can still disagree about which version is current. When a user reloads and sees yesterday's picture, is the system broken for answering fast, or correct for refusing to guess?