Loading...
Loading...
Poison pill isolation, retry exhaustion handling, and DLQ monitoring
The outbox plus idempotent consumer made redelivery harmless: replays land as no-ops. Harmless is not the same as eventually successful. Think of a clinic waiting room as the single analogy for this whole piece: most patients move through quickly, but one patient with a condition nobody can diagnose keeps being sent back to the front of the line, and everyone behind waits. In a queue that looks like one malformed event with broken JSON, which is text shaped like data but with a syntax error, or an unexpected empty value where the code assumed a name. The consumer, which is the worker program pulling messages, throws an error, sends a negative acknowledgment, which is a signal telling the broker, meaning the middleman service holding messages, that the message was not processed, and the broker redelivers it. The worker throws again, and the cycle repeats forever while healthy messages behind it starve.
The naive fix is to retry forever, on the theory that the failure is temporary and the next attempt will succeed. One team ran that policy with a fixed 5-second delay and measured it: a single poison message redelivered 17,280 times in one day, each attempt burning a full handler run plus an error log line, while the healthy messages behind it aged past every latency objective. That works when the dependency recovers in seconds, and it fails completely for poison messages, which are messages that can never succeed no matter how often they are retried, because the content itself is broken. Infinite retry turns one poison message into a head-of-line block, which is a jam where the first item stops the entire line, and burns your error budget on work that was doomed from the first attempt.
Queue: [good][good][POISON][good] -> consumer picks POISON try -> exception -> negative-ack -> redeliver -> exception -> repeat healthy messages behind POISON wait while the same failure loops
The real solution is a dead-letter queue, which is a separate quarantine queue where messages go after they have failed too many times, instead of returning to the main line. Triage, which is sorting failures by cause the way a clinic sorts patients, happens there, calmly and with evidence attached, while the main queue keeps moving.
Attempt 3 to 5 times with exponential delay, which is waiting progressively longer between attempts such as 2 seconds, then 8, then 30. On exhaustion, route to the quarantine queue instead of requeuing into the main line.
try { handle(msg) }
catch(e) {
if (msg.redelivery >= 3) dlq.send(msg, {error:e})
else nq.nackWithDelay(msg, backoff(msg.redelivery))
}Headers, which are small key-value labels traveling with a message, should carry the original queue name, the error text, the attempt count, and the first-seen timestamp, so anyone reading the quarantine knows what broke without replaying it blind.
Alert when quarantine depth rises above zero, group the quarantined messages by error type on a dashboard, and replay, which is re-feeding quarantined messages to the fixed consumer, only after the root cause is fixed, reusing the same idempotency key, which is a stable identifier that lets the handler recognize a replay as already done.
Your image-resize worker fails because the upstream thumbnail service is down for twenty minutes. Without bounded retries, that message redelivers thousands of times, burning compute and paging everyone. With deliberate numbers, it backs off quietly, parks in quarantine, and replays cleanly after the dependency recovers. Visibility timeout, which is the period a broker hides a message after delivering it so no other worker grabs it meanwhile, must exceed your slowest realistic handler time, or slow work reappears while still running and gets processed twice.
retry budget (typical starting point):
maxReceiveCount: 4 # 1 initial + 3 retries, then quarantine
backoff: 2s, 8s, 30s # exponential with jitter +-20%
visibility: 90s # must exceed p99 handler (here ~45s)
alert thresholds:
quarantine depth > 0 for > 10 min -> notify owner channel
quarantine depth > 100 or > 1% of flow -> page (format bug likely)
age of oldest quarantined msg > 4h -> replay runbook overdueJitter, which is random variation added to each delay, matters because a thousand failures retrying on exactly 30-second boundaries hammer the recovering dependency in lockstep the moment it returns. Adding plus-or-minus 20 percent smears retries across time, so the dependency sees a ramp instead of a wall.
Timeouts and 503 responses, which mean the server is temporarily unavailable, deserve backoff and replay because they may succeed later. Format violations and parse errors will never succeed, so route them to quarantine on the first attempt and alert immediately instead of burning three retries proving what the first error already told you.
Every broker offers quarantine, but each names it differently and each has a trap waiting. SQS, which is a hosted queue service where you pay per call, moves messages automatically once configured. RabbitMQ, which is a self-run broker with rich routing, re-publishes through a side exchange you must declare. Kafka, which is a distributed log kept on disk, has no built-in quarantine at all, so your consumer code must produce the failed record to a separate topic itself.
SQS: redrive policy on source queue
deadLetterTargetArn: arn:aws:sqs:...:orders-dlq
maxReceiveCount: 4 # ReceiveCount > 4 -> moved automatically
RabbitMQ: dead-letter exchange at queue declare
x-dead-letter-exchange: orders.dlx
x-dead-letter-routing-key: orders.dlq
x-message-ttl / max-length can also route overflow to DLX
Kafka: no built-in DLQ — build it in the consumer
on exhausted retries: produce to orders-dlq (same key + headers)
headers: x-original-topic, x-error, x-attempts, x-first-seen
DLQ topic: more partitions than source, 30-day retention for forensicsYou fixed the empty-customer bug and the quarantine holds 40,000 orders. Replaying all of them at full speed hits the fixed consumer with a week of backlog plus live traffic at once: it falls over, everything lands back in quarantine, and the replay of the replay fails. The edge cases multiply when ordering matters, because replaying out of order can apply a cancellation before the order it cancels, and when the original handler partially succeeded before crashing, because a naive replay repeats a side effect like a second charge.
The quarantine turns mystery failures into a to-do list with evidence attached. But quarantine is a human pushing the replay button: it never retries the welcome email in ten minutes on its own, never wakes the digest at 8am, and never answers whether the export finished. The layer that does all of that choreography without a human in the loop has its own name.