Kafka vs RabbitMQ: Which Message Broker Should You Choose?

RabbitMQ: The Broker Does the Work

RabbitMQ implements the AMQP protocol. A producer publishes a message to an exchange, not directly to a queue. The exchange examines the message's routing key and, based on its type and configured bindings, routes the message to one or more queues. Consumers subscribe to queues and pull messages from them.

RabbitMQ message flow:

 Producer ──► [Exchange] ──(routing key: "order.created")──► [Queue A] ──► Consumer 1
                   │                                                        (processes, sends ACK)
                   └──(routing key: "order.*")──────────────► [Queue B] ──► Consumer 2
                                                                            (processes, sends ACK)
                                                              
 After ACK: message is deleted from the queue.

The exchange types determine routing behavior:

• Direct — routes to queues where the binding key exactly matches the routing key. One message goes to one queue.
• Topic — routes based on wildcard pattern matching (order.* matches order.created and order.cancelled).
• Fanout — ignores the routing key entirely and broadcasts to every bound queue. Useful for pub/sub.
• Headers — routes based on message header attributes instead of the routing key.

After a consumer processes a message, it sends an acknowledgement (ACK) back to RabbitMQ. The broker then deletes the message from the queue. If the consumer crashes before sending the ACK, the message is requeued and delivered to another consumer. This gives you at-least-once delivery by default.

If a message fails processing repeatedly, you can route it to a dead letter exchange (DLX), which pushes it into a separate queue for inspection. This prevents poison-pill messages from blocking the main queue indefinitely.

Kafka: The Log Does the Work

Kafka stores messages in an append-only, immutable log. A producer writes a message to a topic. Each topic is divided into partitions — ordered sequences of messages stored on disk. Within a partition, every message is assigned a sequential offset (0, 1, 2, 3...).

Consumers don't receive messages pushed from the broker. They pull from the log by specifying which offset to read from. The broker does not track which messages have been "consumed" — the consumer is responsible for maintaining its own position.

Kafka topic with 3 partitions:

 Partition 0: [ msg0 | msg1 | msg2 | msg3 | msg4 | ... ]
                                              ↑ Consumer A (offset 4)

 Partition 1: [ msg0 | msg1 | msg2 | msg3 | ... ]
                               ↑ Consumer B (offset 2)

 Partition 2: [ msg0 | msg1 | msg2 | ... ]
                        ↑ Consumer C (offset 1)

 Messages are NOT deleted after reading.
 Retention policy (e.g., 7 days) determines when old segments are purged.

Because messages are not deleted after consumption, multiple independent consumers can read the same topic at different speeds. A real-time analytics service might be reading at offset 50,000 while a batch ETL job is still at offset 12,000. They don't interfere with each other.

Consumers are organized into consumer groups. Within a group, each partition is assigned to exactly one consumer. This is how Kafka achieves parallel processing: if you have 6 partitions and 6 consumers in a group, each consumer reads from one partition. Adding a 7th consumer does nothing — it sits idle, because there are only 6 partitions to assign.

Why Kafka is fast

Kafka achieves high throughput through three mechanisms that work with the operating system rather than against it:

• Sequential disk writes. Appending to the end of a file is fast — the disk head doesn't need to seek. On modern SSDs, sequential writes can saturate the disk's bandwidth. On spinning disks, sequential I/O is orders of magnitude faster than random I/O.
• OS page cache. Kafka delegates caching to the operating system. Recently written messages stay in the kernel's page cache and can be read by consumers without hitting disk at all. This avoids duplicating data in JVM heap memory.
• Zero-copy transfer. When a consumer reads data, Kafka uses the OS's sendfile() system call to transfer data directly from the page cache to the network socket, without copying it into application memory. This reduces CPU overhead significantly under high read throughput.

Ordering Guarantees

This is where the architectural difference has practical consequences.

Kafka guarantees message ordering within a partition. Messages sent to the same partition are stored and delivered in exactly the order they were produced. There is no ordering guarantee across partitions. To ensure all events for a specific entity (say, all actions foruser_id=42) arrive in order, you use that entity's ID as the partition key. Kafka hashes the key to deterministically assign it to a partition.

RabbitMQ guarantees FIFO ordering within a queue, but only if a single consumer is reading from it. With multiple competing consumers on the same queue, ordering is lost: a faster consumer may finish processing message #5 before a slower consumer finishes message #4. If a message is nacked and requeued, it further disrupts the sequence.

Replay and Retention

This is the sharpest difference in practice.

In RabbitMQ, once a message is acknowledged, it's gone. If you deploy a buggy consumer that processes messages incorrectly, you cannot go back and reprocess them. The data no longer exists in the broker.

In Kafka, messages are retained according to a configurable policy (e.g., 7 days, or until the topic exceeds a size limit). A consumer can reset its offset to any point in the retained history and reprocess messages. This is valuable for:

• Recovering from consumer bugs — fix the bug, reset the offset, reprocess
• Adding a new consumer that needs to process historical data from the beginning
• Event sourcing architectures where the log is the source of truth
• Running different analytics queries over the same stream of events

When People Choose Wrong

Using Kafka as a simple job queue

If you have a web server that needs to offload image resizing to background workers, and each job is independent, and you don't need replay, and you want the broker to handle routing — Kafka is overkill. You inherit the operational complexity of managing a Kafka cluster (broker configuration, partition management, consumer group coordination, potentially ZooKeeper or KRaft) for a problem that a single RabbitMQ instance or even a Redis list solves cleanly.

Using RabbitMQ as an event stream

If you need multiple independent services to read the same events, at their own pace, with the ability to replay history — RabbitMQ fights you at every step. You can set up fanout exchanges to duplicate messages to multiple queues, but each queue is an independent copy of the data. There is no shared log, no offset management, and no replay after acknowledgement. You end up building increasingly complex workarounds for something Kafka handles natively.

Throughput

Kafka is designed for throughput measured in millions of messages per second per cluster. Its architecture — sequential writes, zero-copy reads, batching, and partition-level parallelism — is optimized for moving large volumes of data with low per-message overhead.

RabbitMQ optimizes for per-message features: routing, priority, TTL, acknowledgement tracking. These features add overhead per message. RabbitMQ typically handles tens of thousands of messages per second per node — which is plenty for most applications, but is a different operating point than Kafka.

Raw throughput numbers depend heavily on message size, replication factor, acknowledgement settings, hardware, and network. Quoting specific numbers without those variables is misleading. The architectural point is that Kafka trades routing flexibility for throughput, and RabbitMQ trades throughput for routing flexibility.