Delivery Semantics

The letter that might have arrived twice, or never

The mailbox-or-diary choice settled who reads and for how long. It said nothing about the crash in the middle of the handoff. Think of delivery promises like postal mail, the single analogy for this piece: the post office can promise to never hold your letter back, to keep trying until it arrives even if that means two copies, or to somehow achieve exactly one arrival, and each promise costs a different amount of tracking. You send an email job to a worker, the worker sends the actual email through SMTP, which is the protocol servers use to hand email to each other, and then crashes before acknowledging, which is telling the broker, meaning the middleman holding the message, that the work finished. Did the email go out? Redeliver and the user gets two emails; do not redeliver and they get none. The network makes it impossible to know without extra coordination, because a timeout tells you the acknowledgment never arrived but never tells you whether the work happened before the crash, so every system picks a different guarantee at a different price.

The naive fix is to assume the infrastructure handles it, meaning retries plus acknowledgments somehow add up to exactly once. That fails because the uncertainty sits in the gap between doing the work and recording that you did it: no amount of retrying removes the crash window between a sent email and a recorded acknowledgment. The real solution is to choose explicitly which side of the trade you want, and then make the dangerous side harmless through design.

Producer -> Broker -> Consumer
   ack?       ack?      ack?
Timeout anywhere -> did it commit? -> retry -> duplicate or loss

Three delivery promises, each with a different scar

CREATE TABLE processed(id TEXT PRIMARY KEY);
-- handler
INSERT INTO processed VALUES ($msgId) ON CONFLICT DO NOTHING;
IF inserted THEN process();

How a forty-nine-dollar charge happens twice

A payments worker pulls an order to charge 49 dollars from SQS, which is a hosted queue service where you pay per call instead of running brokers, calls the payment gateway, and the container is killed before the message is deleted. SQS redelivers after the visibility timeout expires. The retry charges 49 dollars again, support refunds one charge, the ledger shows two, and finance asks why the dashboard counted revenue twice. This is at-least-once in the wild: redelivery is guaranteed by the infrastructure, and uniqueness is your job in the handler.

Idempotency keys make redelivery harmless by design

You cannot prevent redelivery, so you make redelivery harmless. Every message carries a stable idempotency key, which is an identifier derived from the business operation such as the order id plus the word charge, never a fresh random value per attempt, and the handler inserts it atomically with the business write inside one transaction. The same key twice means already done, and the database enforces it with a uniqueness constraint even when two workers race on the same message.

-- Postgres: business row + dedup in ONE transaction
BEGIN;
  INSERT INTO idempotency_keys(key, created_at)
  VALUES ('order-9917-charge', now())
  ON CONFLICT (key) DO NOTHING
  RETURNING key;
  -- if no row returned: duplicate, ROLLBACK and ack without work
  UPDATE orders SET status='charged', charged_at=now()
  WHERE id='order-9917' AND status='pending';
COMMIT;

-- Payment API equivalent:
-- POST /v1/charges with header Idempotency-Key: order-9917-charge
-- gateway dedups for 24h; safe to retry on timeout

The outbox collapses two risky writes into one safe commit

Your service updates a Postgres row and publishes an order-paid event to Kafka, which is a distributed log kept on disk for replay. Those are two writes with no shared transaction, so a crash between them leaves the database saying paid while downstream never hears it, or downstream hearing paid while the database says pending. The outbox, which is a table inside your own database where you stage outgoing events, fixes it by collapsing two writes into one: write the business row and the event row in the same local transaction, then let a relay process publish the event row to the log. Consumers deduplicate on the event id, so relay replays stay harmless.

-- same transaction: state + intent
BEGIN;
  UPDATE orders SET status='paid' WHERE id='order-9917';
  INSERT INTO outbox(event_id, aggregate, type, payload)
  VALUES ('evt-9917-paid','order-9917','OrderPaid','{"total":49}');
COMMIT;
-- relay tails outbox -> publishes to Kafka, marks sent
-- consumer dedups on event_id; safe to replay relay output

Choose the promise by asking what scares you most

Duplicates handled, losses bounded, costs understood. But guarantees assume the workers stay up, and some messages are born to fail. What happens to the ones that never succeed, no matter how often you retry?