Loading...
Loading...
RabbitMQ vs Kafka — routing queues vs replayable logs for beginners
The edge layer you just left serves stale pages for 60 seconds while origin recovers, purged precisely by surrogate key. That trick dies the moment the bytes are not a cacheable page but an order that inventory, email, and analytics must each act on exactly as it happened. If checkout calls them directly and email is down, checkout fails too. Decoupling, which means placing a middleman between the sender and the receivers so neither waits on the other, solves it, but the handoff can be a queue that routes each message to one worker and then deletes it, or a log that appends every event to an ordered sequence and lets readers replay it. Think of a mailbox that empties against a diary that keeps every page, the single analogy here: the mailbox hands each letter to one reader and throws it away, while the diary keeps every entry so late readers can catch up from page one.
The naive fix is to treat them as interchangeable, since both sit between services and both hold bytes. That fails the moment requirements diverge: the team that needs replay discovers the queue deleted the history, and the team that needed one worker per task discovers five log readers each processed the same payment. The real solution is to match the retention model, meaning whether data disappears on acknowledgment or persists by time, to the consumption model, meaning whether each record goes to one worker or to many independent readers.
One team tried refusing the choice by writing every event to both: RabbitMQ for the workers, Kafka for analytics. At 500,000 mobile events per second that doubled produce traffic to a million writes per second and doubled the disk math below, while the two systems disagreed on order, because queue delivery order and log offset order are different sequences. Every reconciliation job became a cross-system join on timestamps that never quite matched. Two piles cost twice the hardware and still answer ordering questions differently, so the choice stands.
| Concern | Queue wins | Log wins |
|---|---|---|
| Routing | Topic to queue with routing keys and per-message priority decided by the broker | Partition by key only, meaning records with the same key land together; routing logic lives in the consumer |
| Ordering | Per-queue order with a single consumer, no order with parallel competing consumers | Order within each partition guaranteed, total order across partitions sacrificed for parallelism |
| Recovery | Requeue on negative acknowledgment, delayed retry through the dead-letter exchange | Seek to an earlier offset, which is rewinding the bookmark, and reprocess history after a bugfix |
| Scale | Add consumers as competing readers on the same queue | Add partitions for parallelism, but existing keys do not redistribute automatically |
Picture the analytics team arriving two weeks after launch asking to replay every order event since day one to rebuild a funnel. With a queue that history is gone, because acknowledged means deleted. With a log the answer is arithmetic: ingress rate times retention time times copy count. Get the math wrong and the log either fills its disks or silently drops the replay window you already promised.
retention math: 12 partitions x 7 days x 5 MB/s ingress
= 5 MB/s x 604,800 s = ~3 TB retained cluster-wide
per broker (3 brokers, replication factor 3): ~3 TB total on disk per broker
queue math: depth x avg size = memory pressure
200,000 msgs x 4 KB = 800 MB sitting unacked — page before thisKeep retention at 7 days minimum so replays survive a normal bugfix cycle, and size disks for retention multiplied by peak throughput multiplied by replication factor, which is how many copies of each record are stored. Size brokers by sustained megabytes per second per partition, roughly 10 to 50, rather than by message count, since bytes fill disks and network, not counts.
Alert on queue depth, which is messages waiting, and on oldest-message age together. Depth above 10 times what one consumer drains per minute, or oldest age past 5 minutes for a latency-sensitive queue, means consumers are falling behind rather than absorbing a spike.
A tuned queue broker does tens of thousands of messages per second with per-message routing, while a log does hundreds of thousands to millions per second per cluster using sequential disk writes and zero-copy reads, which are reads that send bytes without copying them through user memory. If you need 500,000 events per second from mobile telemetry, that number alone picks the log.
Your on-call story splits completely. Queues die from poison and pileup; logs die from lag and rebalancing, which is the log reassigning sections when readers join or leave. Knowing which alert belongs to which system saves an hour of misdiagnosis.
One malformed order with an empty customer field crashes the only consumer, which negatively acknowledges, which redelivers, forever, while healthy messages behind it starve. The fix is a dead-letter queue, which is a quarantine queue for repeat failures, after 3 to 5 receives: without it a single bad byte halts the whole line. The sibling failure is a visibility timeout, which is how long a delivered message stays hidden, set too short, so a slow handler looks dead, the message reappears, and two workers process the same order twice.
A consumer falls 2 million offsets, which are positions in the sequence, behind after a bad deploy. It now needs hours to catch up, fetching old segments while new writes keep arriving, so lag grows faster than it drains. Then someone adds consumers to help, triggering a rebalance that pauses every section for seconds and pushes lag further. The cure is autoscaling consumers on lag plus cooperative rebalancing, which moves sections incrementally instead of stopping everything at once, never a full stop-the-world reassignment.
You deploy checkout events to a topic, which is a named stream in a log, created with one partition and weak acknowledgments. Launch-day writes vanish when a broker restarts, and one consumer saturates while two brokers idle. Three dials would have prevented it: partitions, which set the parallelism ceiling, acknowledgments plus in-sync replicas, which set the durability contract, and retention plus visibility, which set how long data survives and how duplicates behave.
Kafka topic "orders" (sane starting point):
partitions: 12 # parallelism ceiling = partition count
replication.factor: 3 # survive 1-2 broker losses
min.insync.replicas: 2 # writes fail rather than go undurable
acks: all # leader + in-sync replicas confirm
retention.ms: 604800000 # 7 days — replay window, not ack-driven
compression.type: lz4
SQS queue "orders" (sane starting point):
VisibilityTimeout: 60s # > p99 handler time, else duplicates
maxReceiveCount: 4 # then -> dead-letter queue
DelaySeconds: 0, MessageRetentionPeriod: 4 days
RabbitMQ consumer:
channel.basic_qos(prefetch=10) # one worker, 10 unacked max| Knob | Too low | Too high |
|---|---|---|
| Partitions | One hot broker carries everything with no parallelism | Slow rebalances with thousands of open file handles |
| Visibility timeout | Duplicates, because slow work looks dead and reappears | Slow recovery, because crashed work hides until the timeout expires |
| Retention | Replay window vanishes before the bugfix ships | Disks fill and old segments cost real money |
Structure picked. Now the harder contract: when a machine dies between sent and received, what may vanish, what may double, and what you must build so neither ruins anyone's day.