Content Negotiation, ETags & Conditional Requests

Safe retries still re-download. Validators stop the waste

The last topic made repeated calls safe with idempotency keys, propagated deadlines, jittered backoff, and breakers, then ended on the remaining waste: a million clients re-fetching an unchanged 20KB payload still burn 20GB a day. The client already has product.json v12 in its HTTP cache, the browser's local store of past responses. Fetching it again costs 20KB and a trip to the origin server. Conditional requests let the server say “not modified, reuse what you have” with a 304 status, meaning not-modified with an empty body, and zero body bytes. That requires two pieces: negotiating what representation to cache, and validating it cheaply. An ETag, a version string the server attaches to each response, is the validator the client sends back to ask “still current?”

First fetch: GET /product/42  Accept: application/json
           → 200 ETag: "a3f5"  Cache-Control: max-age=60  body 20KB (cached)

Second fetch: GET /product/42  If-None-Match: "a3f5"
            → 304 Not Modified  Age: 0  (0 bytes) ✓ revalidates

Negotiation and validators

GET /feed HTTP/1.1
Accept: application/json
Accept-Language: en

HTTP/1.1 200 OK
Content-Type: application/json
Vary: Accept, Accept-Language

PUT /doc/42 HTTP/1.1
If-Match: "a3f5"

→ 412 Precondition Failed if someone else wrote first

Count the bytes you never had to send

Conditional requests are bandwidth arithmetic. A 20KB product payload fetched a million times a day is 20GB of origin traffic; at a 90% revalidation rate where 304 responses cost under 200 bytes of headers, daily traffic drops to roughly 2GB of full bodies plus 180MB of headers, over 85% savings. The origin CPU savings run parallel: a 304 short-circuits serialization, templating, and often the database read entirely, answering from a version comparison that costs microseconds.

The knobs are the cache lifetimes around the validators. Short-lived, hot data (feeds, prices) gets max-age=30–60, stale-while-revalidate=30 so browsers revalidate often but never block on it. Versioned static assets with content hashes in filenames get max-age=31536000, immutable and skip revalidation for a year. API responses in the middle land nearmax-age=60, s-maxage=300, s-maxage being the lifetime shared caches like CDNs use instead of the browser's, with ETags, fresh enough for users, friendly enough for the edge.

When the cache key quietly multiplies

Negotiation breaks through key explosion. Every value listed inVary multiplies the cache key space: vary on Accept-Encoding (gzip, br, identity) times Accept-Language (20 languages) times User-Agent fragments and one URI becomes hundreds of cache entries, each with a miserable hit rate. On-call sees edge hit ratios stuck at 30% despite “everything being cached”, because everything is cached sixty times over, once per variant nobody else requests.

Weak validators add a subtler lie. A weak ETag (W/"abc") promises semantic equivalence, not byte identity, fine for a 304, fatal for a ranged resume or an If-Match guarded write, where a single differing byte corrupts the result. And Last-Modified timestamps at one-second granularity miss rapid successive writes entirely while drifting across servers with skewed clocks.

How teams generate ETags without shooting themselves

Strong ETags come from content, not randomness: a hash of the response bytes, a row version column bumped on every write, or a content-addressed asset digest. Random-per-response ETags guarantee every conditional request misses, the validator never repeats, so 304 never fires and the whole mechanism is theater. Generation cost matters too: hashing a 5MB body per request to save a 5MB body nets nothing, which is why large objects use version metadata instead of byte-hashing on the hot path.

Strong: ETag: "v42-a3f5"      ← version + content hash, stable until bytes change
Weak:   ETag: W/"v42"           ← semantically same, bytes may differ (no byte-range use)
Broken: ETag: "req-8f2a1"       ← random per response, 304 never happens ✗

The lost-update story If-Match was invented for

Two editors open the same document at version 12. Editor A saves first, now the server holds version 13. Editor B, still looking at version 12, hits save: without a precondition, B's stale write silently overwrites A's fresh one, and nobody is told a thing. WithIf-Match: "v12" on every write, B's request fails with 412 the moment versions diverge, the client refetches version 13, merges, and retries. Optimistic concurrency without a single lock held, at the cost of one header per write.

The knob is granularity: version per document for wikis and settings, per row for database-backed resources, per field only when contention is genuinely field-level. Coarse versions cause false conflicts (two editors touching different paragraphs still collide); fine versions cost bookkeeping nobody reads. Per-resource versions with 412-plus-retry handling cover nearly every collaborative-editing design you will run into.

The contract your headers sign with every CDN

ETags and negotiation headers do not stop at the browser, every CDN, a network of cache servers spread near users, between you and the user replays the same protocol. The edge keys on the URI plus Vary values, revalidates with If-None-Match upstream, and serves 304s downstream from its own stored validators. A missing Vary on Accept-Encoding at origin therefore poisons thousands of edge locations at once: the first client's gzip bytes get served to a client that cannot decompress them. Header correctness at origin is edge correctness at scale, which is why CDN debugging almost always starts with curling origin headers directly.

curl -I origin/product/42                    → check Cache-Control, ETag, Vary
curl -H "Accept-Encoding: gzip" origin/…    → must differ or Vary must list it
curl -H 'If-None-Match: "a3f5"' origin/…    → expect 304, else validator is broken

Compression is negotiation too, and the savings dwarf the rest

Accept-Encoding negotiation routinely shrinks text payloads 3–5×: a 100KB JSON feed leaves the server near 20KB over Brotli, worth more than every 304 on the page combined for first-time visitors. The failure mode is compressing what is already compressed, gzip on JPEGs or Brotli on videos burns CPU for single-digit percentage gains while adding latency, and compressing tiny responses where the framing overhead exceeds the savings. The sane line: compress text over ~1KB, pre-compress static assets at build time, never touch media bytes.

BREACH-style attacks, a compression side-channel where attacker-influenced input sharing the compression window with a secret lets the secret be read back, are the sharp edge worth handling with care: compression plus encryption can leak secrets when attacker-influenced input shares the compression window with a token. The standard mitigation is refusing to compress responses containing secrets on pages that reflect user input, one more reason compression policy lives at the edge proxy with explicit content-type allowlists, not as a global toggle.

The mobile app that caches forever and revalidates never

Native apps are the worst HTTP citizens: they cache aggressively in custom stores, ignore Cache-Control nuances, and ship versions you cannot force-upgrade , a stale product list baked into app version 4.2 keeps hitting an old response shape for months. Teams that depend on header revalidation alone discover their oldest clients never send a conditional request at all. Versioned endpoints (v1, v2) plus short server-side TTLs bound the damage: old apps get frozen but correct responses, new apps get fresh ones, and nobody's cache decides the API contract.

The validator habit that replaces re-downloading

Negotiate representation once, validate cheaply forever, ETags with If-None-Match, the header carrying the stored version back for comparison, for 304s, Vary kept to the smallest key that stays correct, and If-Match, the header carrying the expected version on writes, for conflict-free writes. Every variant negotiated multiplies cache entries and divides hit rate, so normalize high-cardinality inputs before they reach the key and reserve strong validators for the bytes that must be exact. One consistency detail earns its keep here: when two servers generate different ETags for identical content, say from server-local data like inode numbers, the filesystem's internal file identifiers, a client validated against server A sends If-None-Match to server B, comparison fails, and the client re-downloads bytes it already holds. Derive ETags from shared state such as a version column or content hash instead.

When a middlebox eats your validators

The ugliest 304 failures involve nobody's code: a corporate proxy strips ETag headers, a misconfigured CDN collapses If-None-Match, or a framework regenerates validators per response, and revalidation silently stops working for a slice of users. The symptom is origin traffic that will not drop no matter how correct the headers look in your curl tests, because curl, the command-line HTTP tool, is not behind the offending middlebox, a proxy between client and server that rewrites traffic in transit. Log 304 rates per client population, not just globally; a cohort at 0% revalidation while everyone else sits at 90% is the fingerprint of a validator-eating intermediary, and the fix is end-to-end header auditing, not more TTL tuning.

The shortcut teams try first: timestamps instead of content hashes

Before hashing content, teams validate with Last-Modified timestamps alone, because every filesystem gives them away for free. It fails on arithmetic. One-second granularity misses rapid successive writes entirely, so two edits inside the same second compare equal and the second silently never ships, while clocks skewed minutes apart across servers disagree about which copy is newer. Content hashes and version columns cost one bump per write and compare exactly; timestamps compare approximately, and approximately is how users read yesterday's price with today's date on it.

The three habits that replace re-downloading

Negotiate representation once, validate cheaply forever, and compress text over 1KB: bandwidth falls severalfold while origins answer version comparisons instead of full bodies. But every revalidation that misses still lands somewhere, on a store that must answer concurrent writes with auditable truth instead of approximately-right bytes. The open question is what holds that truth on one machine, with relationships declared instead of reconstructed by hand.