Caching Fundamentals

The specialty store earns its keep on the first hot query

The last module ended by defaulting to the general knife and earning each specialty tool with measured pain, starting with the key-value store already sitting wherever sessions and caches live. This topic is that store's first job. Your homepage shows the ten trending posts. Every visit runs the same expensive query: join, sort, rank, return. A thousand visitors a minute means a thousand identical queries a minute, each taking 80 milliseconds, each holding a connection, each answer identical to the last. The database isn’t doing real work anymore. It repeats the same answer at full price.

A cache remembers the answer. First visitor pays full price; the next 999 get the remembered copy in about a millisecond. Same page, a hundred times faster, database barely involved.

The detour that skips the database

Four wins for the price of one sticky note

What earns a sticky note, and what doesn’t

The two things that can happen, and what each costs

Where do the sticky notes live?

Hit-rate math decides whether the cache pays rent

Effective latency is a weighted average: hit rate × hit cost + miss rate × miss cost. At 1ms hits and 100ms misses, a 90% hit rate means 0.9 × 1 + 0.1 × 100 ≈ 11ms average, nine times faster than no cache. Drop to 50% and the average is ~50ms: still better, but now half your database load never left and every other user pays full price. Below ~80% on a database-protection cache, you are paying for memory, operations, and invalidation bugs in exchange for modest relief.

Database load falls by the same fraction: a 90% hit rate turns 10,000 queries per second into 1,000. That is the difference between a primary gasping at connection limits and one cruising at 20% CPU. Size the cache from the working set, not vibes, if hot keys total 2GB and you provision 512MB, evictions churn constantly and the hit rate ceiling is set by arithmetic, not tuning. Redis memory near 70–80% full with an eviction policy like allkeys-lru, which discards the least-recently-used keys first, is the usual healthy band; past it, every insert evicts something warm.

The morning the cache woke up empty

Cold starts are the failure mode nobody load-tests. Deploy a fleet restart, flush Redis for a migration, or lose a cache node, and 10,000 rps of previously cached reads land on the database within seconds. Connections exhaust, p99, the slowest 1% of requests, climbs from 100ms to seconds, timeouts trigger retries that double the flood, and the load balancer starts evicting app servers that are merely waiting on a drowning database. Users see a full outage caused by a cache event, with the database as innocent second victim.

Warmup discipline prevents it: deploy in rolling waves so only a fraction of cache clients restart at once, keep persistent cache snapshots, Redis RDB, the built-in point-in-time snapshot files it can reload on restart, for fast reload, and consider a brief stale-serving window where expired entries keep answering while fresh values refill in the background. Large chat platforms' public cache postmortems follow this exact arc, the cache is load protection first and latency optimization second, and losing it suddenly is a load event, not a speed event.

The shortcut teams try first: each server remembers alone

Before running shared memory, teams cache inside each app server's own process, because no new infrastructure is needed. It fails on arithmetic. Thirty boxes each holding a 2GB working set burn 60GB of RAM for 2GB of distinct answers, every deploy wipes all thirty copies at once and replays the cold-start flood, and a user landing on a different box after each click misses every time. One shared store serves every box from one copy and survives any single deploy; per-process memory is a speedup for hot keys, not a substitute for the shared copy.

Add a cache when the workload repeats

Because the workload repeats: same hot keys, expensive to compute, tolerant of brief staleness, a 90% hit rate turns 10,000 database reads into 1,000 and drops average latency nearly tenfold. Every cache is a second copy of the truth with its own sizing, expiry, and failure story, so cache only what repeats, size to the working set, and make a cold cache a load event you have rehearsed. One paradox to keep in mind when the numbers look perfect: a 99% hit rate with an overloaded database means the 1% that misses is disproportionately expensive, uncached analytics scans, wildcard queries, or cache-busting unique keys that each cost 100× a normal query. Hit rate measures count, not cost. Weight misses by database time, find the costly 1%, and either cache it differently or stop letting it reach the primary at all.

One shared store drops average latency from 100ms toward 11ms at a 90% hit rate and turns 10,000 database reads into 1,000. But it fixed one hop: the browser still re-downloads the same logo, the edge still refetches the same stylesheet, and the database still re-runs the same count query, because each layer only remembers its own job. The open question is where else the sticky notes live, cheapest check first, and what each layer catches that the one above missed.