Apache Kafka

Partitions decide ordering and parallelism

A partition, which is one ordered section of a topic, guarantees order only inside itself, never across partitions. If order matters for one user timeline, key every event by user id so all events for one user hash to the same partition. Scaling a topic means adding partitions, but you cannot reduce them later and re-keying changes which records land where.

Consumer groups share the reading work

A consumer group, which is a set of workers sharing one logical subscription, splits partitions among its members: a group named fraud-detectors with 3 workers over 3 partitions gives each worker one partition. Adding a fourth worker does nothing until partitions increase, because the maximum useful workers in one group equals the partition count. On join or leave, the group coordinator, which is the broker process assigning partitions to members, rebalances, which is reassigning partitions, using range, round-robin, or cooperative sticky assignment that moves incrementally instead of stopping everything.

Offsets and retention decide what survives

The broker retains records per retention time, keeping the notebook pages for days regardless of who read them. Each consumer commits its offset, which is the next position to read, to an internal offsets topic. Automatic commit before processing duplicates on crash, so prefer manual commit after handling completes.

consumer.process(record){
  await handle(record);
  await consumer.commit(record.offset+1); // exactly at record boundary
}

Replicas and acknowledgments decide durability

In-sync replicas, which are follower copies confirmed caught up with the leader, must acknowledge for durable writes. Acknowledging on leader receipt alone is fast but lossy if the leader crashes before replicating, while requiring all in-sync replicas is durable but slower. An idempotent producer, which is a sender the broker deduplicates by sequence number, plus transactions gives effectively-once delivery without duplicates on retry.

Partition count is a parallelism ceiling

Too few partitions means one broker goes hot with no room to spread load, while too many means thousands of open file handles and slow rebalances. Start 6 to 12 per topic and benchmark producer throughput per partition at roughly 10 to 50 megabytes per second sustained.

Watch replicas, lag, and queues together

Watch in-sync replica shrink, which is followers falling behind, plus under-replicated partitions, consumer lag, which is how far the reader bookmark trails the newest page, and broker request queues. Alert on lag past twice your slowest realistic processing time rather than on raw depth alone.

Exactly-once is a deliberate purchase

An idempotent producer plus a transactional id plus consumers reading only committed data gives read-process-write effectively once. It costs latency and operational care, so enable it only where duplicates truly break correctness such as money movement.

Break the loop by shrinking work and widening deadlines

• Raise the maximum poll interval above the slowest realistic batch time, and lower records per poll to shrink each batch.
• Use the cooperative sticky assignor, which moves partitions incrementally, so rebalances avoid stopping every reader at once.
• Commit offsets incrementally during long batches rather than once at the end, so a revoked partition replays minutes instead of the whole batch.

Replica traps that look like log bugs

• In-sync replicas below the minimum means writes fail by design: add brokers rather than lowering the minimum at 2am.
• Under-replicated partitions with the leader on a dying disk mean fail over deliberately before it fails accidentally.
• A hot partition from one celebrity key means salt the key with a suffix or split the tenant onto its own topic.