Hot Key Mitigation & Stampede Protection

Honest invalidation still stampedes on the famous key

The last topic made single keys honest with delete-on-write, double-deletes, and versioned keys, then ended on the remaining threat: one key read 50,000 times a second expiring everywhere at once. A hot key, one cache entry read far more than all others, plus synchronized expiry can cause a full outage, not load, but coherence. Your homepage key home:feed:en is hit 50k rps. It expires. In the next 10ms, 500 app servers miss at once and thunder onto the database with identical queries. Postgres, an open-source relational database, spikes to 100% CPU, queries queue, timeouts cascade to the balancer. One hot key plus synchronized expiry caused a full outage, not load, but coherence.

t0: cache has key (TTL 60s) → 50k rps served from RAM
t1: expire
t1+1ms: 500 workers MISS → 500 SELECT * WHERE feed='home'
t1+100ms: DB CPU 100%, p95 (slowest 5%) 2s → cache still empty, more misses pile
t1+2s: DB starts returning → cache filled → storm ends, but damage done

Three patterns that actually help

var g singleflight.Group
v, _, _ := g.Do("home:feed", func() (interface{}, error){
  return db.QueryFeed()
})

ttl = 60 + rand(0,10) // seconds
if ttlRemaining < ttl*0.1) {
  backgroundRefresh(key) // non-blocking
}

What not to do

Do the stampede arithmetic before it does you

A hot key concentrates the whole fleet's miss cost into one instant. Fifty thousand requests per second against a key with a 60s TTL means three million served reads per generation, and if even 500 app workers miss simultaneously on expiry, each firing a 200ms feed query, the database absorbs 500 concurrent heavy queries where it normally sees a handful. At ~100ms per query per core, that single synchronized miss demands roughly 50 cores for a full second just to clear the backlog, during which latency climbs, timeouts fire, and retries add a second wave on top of the first.

The Redis layer has its own hotspot math: every key hashes to one slot, one partition of the key space, on one node, so the famous key's 50k rps lands on a single Redis instance no matter how many nodes the cluster has. Adding nodes helps total capacity but not this key, per-key throughput caps near 100–200k ops/sec on typical instances, and past it the cache node itself becomes the queue. That is why the fix stack is local-first: a 5-second local least-recently-used cache absorbing 95% of the 50k rps leaves only ~2,500 rps for Redis, which it handles yawning.

The knobs that desynchronize the herd

When the shield becomes the single point of heat

Coalescing has its own failure signature worth recognizing on-call: the fetcher holding the shared single-flight lock stalls (slow query, garbage-collection pause), and thousands of waiters pile behind one promise instead of thousands of queries hitting the database. If the lock has no timeout, one stuck fetch parks the entire fleet behind it, users see a cliff, not a slope, and every dashboard shows healthy databases with zero traffic. Lock timeouts slightly above p99 plus a stale-while-recompute fallback (serve the expired copy while one worker refreshes) turn that cliff back into a slope.

Local caches diverge the same quiet way layered caches do: 500 app servers each holding a 10-second-old copy means two users can see two generations minutes after a price change if refreshes stagger. Acceptable for feeds and leaderboards, unacceptable for balances, which is why the local layer holds only data whose staleness contract explicitly tolerates seconds of skew.

Spot the famous key before it introduces itself

Hot keys rarely arrive announced, they emerge from a viral post, a front-page listing, or a single enterprise customer's sync job. The detection stack is per-key request metrics (top-K tracking in cache telemetry or a proxy histogram), a miss-rate alert that fires when any single key's miss ratio jumps, and database query-shape monitoring that shows 500 identical statements appearing in the same second. Teams that graph per-key traffic see the celebrity rising over minutes; teams with only fleet aggregates meet it during the incident.

The launch playbook for a key you know will be famous: pre-warm it before the announcement (populate all layers, verify hit ratios), pin a long TTL with background refresh for the event window, and put the per-server local least-recently-used cache layer in place ahead of time rather than deploying it mid-spike. Unplanned fame gets the same treatment retroactively, the first stampede is the monitoring that justifies the permanent defenses.

Coalesce, jitter, localize, in that order

Coalesce concurrent misses so one fetch serves many waiters, jitter TTLs so generations never align, and absorb the steady-state heat in a seconds-long local least-recently-used cache, no single remote layer survives a true hot key alone. Each defense trades a little staleness or complexity for survival under synchronization, so the famous key gets all three while ordinary keys stay on plain TTL.

The permanent celebrity that never cools down

Some keys are not famous for an hour, they are famous forever: the global config flag read on every request, the front-page leaderboard, the default locale bundle. Expiring these on any TTL just schedules a recurring stampede, so they get a different contract: effectively no expiry, updated by explicit invalidation or versioned-key rotation, replicated to every app server's local memory at startup, and refreshed by broadcast rather than by miss. A key with no TTL and a push-based update path cannot stampede, because there is no synchronized moment where every holder drops it at once.

Rehearse the stampede before it rehearses you

Stampede defenses are cheap to claim and easy to misconfigure, so expire keys deliberately: delete the famous key in staging (or in production at 3am with the team watching) and measure what actually happens, how many database queries fire, whether coalescing collapses them to one, how long p99 stays elevated. The drill routinely reveals the lock timeout set longer than the fetch it guards, the local LRU that was never enabled in the deploy config, or the jitter applied to the wrong TTL field. A stampede you schedule teaches; one the internet schedules bills.

Surviving the famous key

Coalesce concurrent misses, jitter TTLs across 54–72 seconds, localize the famous key in a 5-second per-server cache. Those three absorb origin heat inside one region. But users sit a 200ms ocean away, and when every edge point-of-presence, each a small data center near its city's users, misses the same key at once, the globe replays the stampede per continent. The open question is how shared caches stationed near users collapse geography itself, with headers, purges, and revalidation doing for distance what coalescing did for concurrency.