Idempotency, Retries, Timeouts & Backoff

Retries are not free, they create new requests

The last topic ended with per-call deadlines of a few hundred milliseconds, connect timeouts near 100ms, and response deadlines near twice the dependency's p99, the slowest 1% of responses. Deadlines decide when to stop waiting, but not what already happened. Imagine charging a card: your service sends POST /charge, the network times out after 5s, but the payment provider actually processed it. If you retry blindly, the user is charged twice. The network is never reliable, so you must decide up front which operations are safe to retry and how.

Client                        Server (charge service)
  POST /charge {id=42} ──────►  [processes, DB commit succeeds]
          ◄─ timeout (5s) ─────   [response lost]
  RETRY POST /charge {id=42} ─►  [charges again → duplicate] ✗
           ──────────
  With Idempotency-Key:42 ──►  [sees key exists → returns prior result] ✓

How do you retry without charging twice?

An operation is idempotent if doing it N times has the same effect as once. Safe methods (GET, PUT with full replace, DELETE) often are; POST rarely is. For non-idempotent POSTs, carry an Idempotency-Key.

// store key + result for 24h
// unique index on (idempotency_key)
const existing = await db.find(key);
if (existing) return existing.response;

const result = await chargeCard(payload);
await db.insert({ key, result });
return result;

POST /v1/charges HTTP/1.1
Idempotency-Key: 7f8a9c2e-... (UUID v4)
Idempotency-Expiry: 86400

{"amount": 1999, "currency": "usd"}

How long do you wait before giving up?

Every outbound call needs a deadline. No timeout means one slow downstream can hold your entire request thread pool. But too-short timeouts cause retry storms.

Deadline propagation:
  Gateway 1s ──► Service A (300ms left) ──► Service B (80ms left)
  If deadline < 0 → fail fast, do not even call B.
  Go, a language built for servers, carries this with context.WithTimeout(ctx, 300*time.Millisecond)

How do retries avoid stampeding together?

Retry only what is retriable: network errors, 429, the status meaning slow down, 503, the status meaning temporarily unavailable, with Retry-After, the header telling you how long to wait, and timeouts for idempotent requests. Never retry 4xx client-error statuses (except 429 for slow-down and 408 for request-timeout) or non-idempotent POST without a key.

Retry: 0ms 0ms 0ms →
Load spike: ████████
→ thundering herd,
   downstream doubles
   every retry

Retry: 100ms 100ms 100ms
Better but clients sync:
 ███   ███   ███
 still bursty

delay = base * 2^attempt
         + random(0, jitter)
Attempt 0:  50-100ms
Attempt 1: 100-200ms
Attempt 2: 200-400ms
→ decorrelated, smooth

// Exponential backoff with decorrelated jitter (AWS recipe)
async function retry(fn, {retries=3, base=50, cap=1000}={}) {
  let delay = base;
  for (let i=0;i<=retries;i++) {
    try { return await withTimeout(fn(), 800); }
    catch(e) {
      if (i===retries || !isRetriable(e)) throw e;
      await sleep(Math.min(cap, Math.random()*delay*3));
      delay = Math.min(cap, delay*2);
    }
  }
}

The shortcut teams try first: retry immediately, three times, no waiting

Before adding backoff, teams retry the instant a call fails, up to three times, because waiting feels like wasting the user's time. It fails on arithmetic. At 1,000 requests per second with a 10% downstream failure rate, instant retries add roughly 300 bonus requests per second of pure retry load, synchronized onto the dependency that is already struggling, and fixed 100ms intervals keep every client marching in the same bursts instead of spreading out. Exponential growth with random jitter is the fix: delays of 50–100ms, then 100–200ms, then 200–400ms decorrelate the herd, and a 10–20% retry budget caps the multiplication no matter how red the dashboard gets.

The multiplication your retries are hiding

Retry math is multiplication, and it surprises everyone once. One gateway call fans to three services, each configured for 3 retries with no budget: a single slow user request becomes up to 4 attempts × 3 services = 12 downstream calls in the worst case. At 1,000 requests per second with a 10% downstream failure rate, that is roughly 1,000 × 0.10 × 3 extra calls ≈ 300 bonus requests per second of pure retry load, aimed at the dependency that is already struggling. This is how a partial slowdown becomes a full outage: retries add load proportional to failure, exactly when load hurts most.

Budgets convert multiplication into a ceiling. Cap retries at 10–20% of active request volume (client-side throttling) and cap total elapsed time at the caller's remaining deadline, a retry with 20ms of budget left against a 200ms-p99 dependency is not optimism, it is queue pollution. Combine with hedged requests, a second attempt sent when latency passes p95, the point 95% of requests beat, with the first answer winning and the loser cancelled, only on the hottest read paths.

When idempotency keys lie to you

Keys break in three production ways. First, the key scope is wrong: keyed per endpoint instead of per intent, so a retried checkout with a slightly different payload gets a new key and double-charges. Second, the dedup window is too short: 24-hour windows exist because mobile clients retry the next day on bad networks, a 5-minute window replays yesterday's charge as a new one. Third, the check-then-insert races: two identical requests arrive together, both miss the lookup, both charge, both insert. Only a unique constraint on the key column (or equivalent atomic primitive) closes that hole; application-level “look before you leap” never does.

Knobs worth memorizing: dedup window 24 hours for money-moving POSTs, key cardinality one UUID v4, a random unique identifier, per user intent generated client-side, replay returns the exact original status and body (including the original 4xx), and key validation rejects reuse across different payloads with a 422 rather than silently returning the old result for a new intent. A 422 status, meaning the request was understood but semantically invalid, signals the mismatch instead of a silent wrong answer.

The safe-retry habit, end to end

Put idempotency keys on every state-changing POST with atomic dedup, let deadlines propagate downstream, back off exponentially with jitter, and hold a retry budget plus a circuit breaker, a guard that stops calling a dependency whose error rate stays high and fails fast instead, so retries stop when the dependency is bleeding. Every retry trades one user's latency for the whole fleet's load, so retry only idempotent failures with budget remaining, and fail fast the moment the math says the retry cannot help. And when the breaker opens and traffic starts failing fast, close it with a half-open probe: after a sleep window (30s is typical), allow one trial request through, and close only on success, consecutive successes if the dependency flaps. Closing on a timer without a probe re-floods a recovering service with full traffic instantly, which is how breakers oscillate and take turns causing the outage they were installed to prevent.

Timeouts are guesses with consequences on both sides

Every timeout value is a bet about the downstream's latency distribution. Set the read timeout at the downstream's p50 and half your calls fail spuriously, each failure triggering a retry that doubles the load that caused the slowness. Set it at 10× p99 and a hung dependency holds your threads for seconds while users abandon the page. The working rule, read timeout near 2× the dependency's p99, connect timeout near 50–100ms same-region, keeps spurious failures under ~1% while bounding the worst hold time to something your thread pool survives.

Total deadlines must shrink as they travel: a 1-second gateway budget that becomes a fresh 1-second timeout at each of three hops is a 3-second user wait wearing a 1-second costume. Propagate the remaining budget in a header and fail fast when it runs negative, a downstream call with 20ms of budget left against a dependency whose slowest 1% takes 100ms should never be sent. Deadline propagation, where the remaining budget travels in a header the way remote-call frameworks and Go's context package, the standard way Go programs carry cancellation, both implement, is what makes this the default, not the exception.

The retry checklist worth memorizing

Idempotency keys, propagated deadlines, jittered backoff, and breakers make repeated calls safe. But safe is not cheap: a million clients re-fetching an unchanged 20KB product payload still burn 20GB of origin traffic a day, every byte correct and every byte wasted. The open question is how clients avoid re-downloading answers they already hold, asking the server "still current?" instead of "send it again?".