Loading...
Loading...
Handling load when consumers can't keep up with producers
Task queues turned waiting users into a draining pile: twenty workers chew 40 videos a minute while fifty thousand arrive in the hour, and the last uploader waits 1,250 minutes. Now shrink the videos to sensor readings and remove the forgiving human at the end. Imagine a sink where the faucet pours 1,000 cups a second and the drain swallows 100. That is the only analogy in this piece, and it carries the whole idea: your sensors emit 1,000 readings a second, your processor handles 100, and the leftover 900 messages every second have to sit in a queue, which is simply a waiting area in memory or on disk where unread work sits until a worker picks it up. After one hour the pile holds 900 times 3,600, which is about 3.2 million messages, and if each message is even one kilobyte that is over 3 gigabytes of memory gone to backlog alone.
The naive fix is to make the waiting area bigger, which means raising the buffer, the block of memory set aside to absorb bursts, to some huge number and hoping the spike passes. That works when the spike is brief, and it fails completely when the arrival rate stays above the drain rate, because a bounded mismatch in rates always beats a bigger box: the pile still grows by 900 a second, it just takes longer to hit the wall, and when it does the process runs out of memory and dies. Backpressure, which is any signal from the overwhelmed downstream side telling the upstream side to slow down, is the real solution because it turns an invisible pileup into an explicit choice about what to sacrifice.
Faucet wide open
Pile grows forever
Drain at capacity
An unbounded buffer is a slow-motion crash
Walk the arithmetic first: pile growth per second equals arrival rate minus drain rate, and memory needed equals growth times seconds times bytes per message.
Every workable answer sacrifices something different, and the mechanics below show what moves where. The only wrong move is pretending the pile can grow forever:
Dropping, which means deliberately discarding the oldest or newest message when the buffer is full, keeps the drain moving at 100 per second no matter what the faucet does. A metrics dashboard, which is a page showing recent graphs like CPU over the last minutes, does this constantly: last minute's CPU graph matters and the one from Tuesday does not, so losing old points costs nothing. The mechanics are a bounded queue with a drop policy plus a counter of dropped messages, and the edge case is silent data loss that looks healthy: if nobody watches the drop counter, you will report smooth graphs built from missing data.
Blocking, which means refusing to accept new work and making the sender wait until space frees up, loses nothing but moves the slowness upstream: the faucet now runs at the drain's 100 per second because the queue's push operation does not return until a slot opens. In practice this is a bounded channel whose send call parks the sender thread, and the edge case is a chain reaction: if the sender is itself a request handler holding a user connection open, every blocked send holds a connection, and a slow drain can exhaust the upstream connection pool in minutes. It works as long as upstream can wait too, which is true for internal pipelines and payment flows where every item is audited, meaning every item must be accounted for later.
A bounded buffer, which is a waiting room with a fixed capacity such as 10,000 messages, absorbs bursts gracefully because real traffic is spiky and averages lie: a ten-second burst at 1,000 per second against a 100-per-second drain needs only 9,000 slots to ride out without dropping or blocking. The mechanics that matter are that the bound is doing all the work, so when the room fills you still fall back to dropping or blocking, and the edge case is a slow leak that looks like absorption: if the average arrival stays at 150 against a 100 drain, the room buys you exactly 200 seconds at 10,000 slots and then behaves as if it never existed. Buffers buy time, not capacity.
Rate limiting, which means rejecting or delaying senders above a fixed allowance such as 100 requests per second per API key, pushes the decision to the door: rejected work fails fast and loudly with an error the caller can see instead of dying quietly inside a queue. The mechanics are usually a token bucket, which is a counter that refills at the allowed rate and spends one token per request, sitting at the edge before any queue. The edge case is retry amplification: if every rejected caller retries immediately, your 100-per-second cap sees 300 attempts per second of retries plus fresh load, so limits must pair with backoff, which is waiting progressively longer between retries, and a Retry-After hint telling the caller when to come back.
Scaling consumers, which means running more workers so combined drain rate rises from 100 toward 1,000 per second, is the only answer that removes the mismatch instead of managing it, and the only one with a cloud bill attached. The mechanics are autoscaling driven by queue depth, which is the count of waiting messages, and consumer lag, which is how stale the newest processed message is: add one worker per 1,000 queued messages, for example. The edge case is startup delay: new workers take minutes to boot and warm caches while floods take seconds, so scaling alone without shedding load in the meantime still overflows the sink.
Backpressure sounds abstract until you see where you have already configured it. Each of these is the same faucet-and-drain negotiation wearing different clothes:
RabbitMQ, which is a message broker, meaning a middleman service that routes messages between senders and workers, supports prefetch, which is the maximum number of unacknowledged messages one worker may hold.
channel.basic_qos(prefetch=10)A worker holds ten unacknowledged messages at most, and the broker waits instead of pushing an eleventh. That is blocking backpressure built into the protocol.
Kafka, which is a distributed log, meaning an append-only record of events kept on disk that many readers can replay, lets consumers pull rather than be pushed to.
max.poll.records=100Each consumer fetches at most 100 records per poll at its own pace, and the log waits patiently because retention is time-based rather than acknowledgment-based.
Node streams, which are the built-in way one part of a program pipes bytes to another, carry a high-water mark, meaning the buffer size at which the stream signals full.
stream.pipe(dest, { highWaterMark: 16 })When the destination buffer passes 16 objects, the source is paused automatically and resumed when the buffer drains. Backpressure plumbed into the language itself.
SQS, which is a hosted queue service where you pay per use instead of running brokers, paired with Lambda, which is a service that runs your function per batch without servers to manage, limits in-flight work explicitly.
batchSize: 10, maxConcurrency: 5Fifty messages in flight at most. The rest wait in a queue that never fills your process memory because it lives outside your machines.
Queue depth, which is the count of messages waiting right now. A depth rising steadily means the drain already lost, so alert long before memory notices, for example when depth exceeds two minutes of drain capacity.
Enqueue-to-completion latency, which is the time from a message arriving to its work finishing. Users feel this number directly, and Little's law, which says average wait equals depth divided by drain rate, converts depth into this number.
Consumer lag, which is how stale the newest processed message is compared to the newest arrived. Lag growing linearly for fifteen minutes means capacity sits below arrival rate, not that a spike is passing through.
Dropped and rejected counts, which are how many messages you discarded or turned away. If you chose dropping, this number is your honesty metric: divide refusals by arrivals to report the true loss rate alongside the smooth graphs.
Your checkout queue holds 9,000 messages and drains 300 per second, so the last message waits 9,000 divided by 300, which is 30 seconds. That is fine for thumbnail generation and fatal for payment authorization, which is the step where a bank approves a charge within a few seconds before the user gives up. The deeper mechanics are the death spiral this exposes: as wait grows past client timeouts, clients retry, retries add to the arrival rate, the pile grows faster, and waits grow further. A queue that looked merely big becomes a feedback loop.
wait = depth / drain_rate
9,000 msgs / 300 msgs/s = 30s tail wait
alert before the lake forms:
queue depth > 2 min of drain -> warn (autoscale candidate)
depth > 10 min of drain -> page (shed or widen drain now)
consumer lag growth > 0 for 15m -> capacity below arrival, not a spike
p99 enqueue-to-complete > SLO -> users already feel itSLO, which is a service-level objective such as 99.9 percent of checkouts under 500 milliseconds, is the line the last gauge watches. Teams that dislike every sacrifice above sometimes try a sixth answer: no shedding at all, just infinite autoscale. At the faucet-drain ratio here that means growing the drain tenfold for a ten-minute spike, while new workers take three to five minutes to boot and warm caches and the flood takes seconds. The spike is over before the cavalry arrives, and the bill covers ten times the machines for the whole hour. Shedding is the admission that some floods end before help can boot.
CPU looks fine while a queue explodes because workers sit idle-blocked waiting on a slow dependency rather than burning cycles. Scale consumers on depth and lag, for example by adding a worker per 1,000 queued messages, with cooldowns of several minutes so new workers finish booting before you add more and the count stops flapping up and down.
A 10,000-message cap with drop-oldest plus a counter beats unbounded growth that runs out of memory at 3am. Shedding, which means deliberately discarding or refusing low-priority work to protect the rest, is graceful degradation in practice: shed background exports first, keep interactive checkout, and count everything you shed so the loss is visible.
A log consumer, which is a worker reading from an append-only log at its own offset, meaning its bookmark in the sequence, falls 3 million offsets behind after a deploy. Each poll now fetches huge batches, garbage-collection pauses grow because the heap fills with one giant batch, processing slows, and lag grows further. Restarting every consumer to help triggers a rebalance, which is the log reassigning which worker reads which section, and that pauses all sections for seconds while replaying uncommitted offsets, so lag doubles instead of shrinking. This spiral ends more streaming pipelines than any bug in business logic.
Piles survive floods only when some signal decides what gives way, and the gauges above tell you which sacrifice you are actually making. Every answer so far assumes the flood is honest: too many real users arriving too fast, shed politely with a 503 and a Retry-After hint. But rate limits and token buckets cannot tell an honest crowd from ten thousand hijacked devices arriving on purpose. When the faucet itself is hostile, slowing down is the wrong game entirely.