How ChatGPT Runs on Postgres: Scaling to 800 Million Users

With a single primary, every write in the world funnels through one machine. That machine cannot be scaled out — only up. So the entire game is keeping load off it. Their outages kept following the same script: something upstream hiccups — a cache layer fails, a bad deploy fires a storm of writes, a new feature ships an expensive query — and database load spikes. Queries slow down. Clients time out. Clients retry. Retries add more load. The cycle feeds itself until ChatGPT itself degrades.

There is a second, deeper problem with writes in Postgres. It uses a system called MVCC, which is just a fancy way of saying: instead of overwriting a row, Postgres writes a whole new copy of it. Update one field in a row with fifty columns and the database copies all fifty. Under heavy write load, the database drowns in old copies — tables bloat, indexes bloat, and background cleanup (called autovacuum) can't keep up. Postgres is a fantastic reader and a mediocre writer. So OpenAI stopped asking it to write as much.

The single highest-leverage move was also the bluntest: stop writing so much to Postgres. Workloads that can be split across many machines — high-volume, partitionable writes — were migrated to a sharded store (Azure Cosmos DB). New features are no longer allowed to add tables to the Postgres deployment at all. New workloads default to the sharded systems.

They also went after writes nobody needed in the first place: application bugs that wrote the same data twice, writes that could happen lazily instead of immediately, and backfills that were throttled with strict rate limits so a maintenance task could never spike the primary. None of this is glamorous. All of it directly buys headroom on the one machine that can't scale out.

Why Not Just Shard Postgres?

Splitting Postgres itself across many writers would mean rewriting hundreds of application endpoints — months or years of work. Since the workload is overwhelmingly reads, and the optimizations below kept working, sharding stays a "someday, maybe" project. Runway beats rewrites.

Whatever reads absolutely must touch the primary — reads inside write transactions, for example — are kept ruthlessly efficient. Everything else goes to replicas. That separation is also what saves them when the primary itself dies: the vast majority of requests are reads, so if the writer goes down, ChatGPT mostly keeps working. A primary failure drops from a company-wide SEV-0 to "writes are failing, reads are fine."

  Users (reads + writes)
        │
   ┌────┴─────────────────────────┐
   │                              │
   ▼ (writes only)                ▼ (reads)
[Primary] ──streams──► [Replica] [Replica] [Replica] … ×50
   │                     across regions, near users
   └── hot standby (instant takeover)

Two safety nets sit underneath. The primary runs with a hot standby — a replica kept in perfect sync whose only job is to take over within moments if the primary dies. And every region runs multiple replicas with spare capacity, so losing any single replica is a non-event.

A handful of bad queries can do more damage than all legitimate traffic combined. Their worst offender joined twelve tables in one query — and spikes of it had caused past SEVs. The fixes are unglamorous and effective: avoid giant multi-table joins, and when a join is truly needed, split it up and do the combining in application code instead of the database.

A special warning for beginners: much of this bad SQL is written by ORMs — libraries that generate database queries from application code automatically. The SQL they produce can be shockingly wasteful. Always inspect what your ORM actually sends to the database. And set idle_in_transaction_session_timeout so abandoned queries can't sit around blocking cleanup forever.

One feature's traffic spike should never be able to take down another feature. But on shared database machines, that's exactly what happens — a new launch with a wasteful query eats all the CPU and everything else slows down with it. The fix is workload isolation: split traffic into tiers (say, high-priority and low-priority) and route each tier to its own machines.

Think of it like hospital triage versus the gift shop — you don't want gift-shop traffic jammed in the same hallway as the emergency room. Same idea across products, too: one product's surge must never degrade another's.

Every database connection eats memory on the server, and each instance caps out (around 5,000 on their setup). Do the math on a fleet: 10 app servers × 100 connection threads each = 1,000 simultaneous connections before a single query runs. Add a traffic spike and you run out — new requests fail even though the database CPU is fine. This failure mode even has a name: connection storms.

The fix is PgBouncer, a proxy that holds a small pool of real, long-lived connections and lends them out per query. Opening a fresh connection costs around 50ms; reusing one from the pool costs about 5ms. They run it next to the databases in every region (cross-region chatter is slow and holds connections open longer), and they tune its timeouts carefully — a pooler with sloppy timeouts just moves the exhaustion somewhere else.

Most reads never reach Postgres at all — a cache in front answers them. The danger is what happens when the cache suddenly empties: thousands of requests miss at once and stampede the database together. The defense is request coalescing: when many requests miss on the same key, exactly one of them is allowed through to the database. Everyone else waits for the fresh value to land back in the cache. One query instead of ten thousand.

Here's the catch with 50 replicas: the primary has to stream every write to every single one of them. More replicas means more network and CPU burned on the one machine you're trying to protect — and laggier replicas. The way out is cascading replication: instead of the primary talking to all 50 replicas, intermediate replicas relay data downstream, like a chain of messengers. That could take them past a hundred replicas — but failover gets trickier (promoting a middle link must not orphan the chain), so they're testing it carefully before trusting production to it.

Rate Limit Everything

The last line of defense sits at every layer: the application, the pooler, the proxy, and the queries themselves. Caps on traffic per endpoint, no hair-trigger retries (short retries during an outage are gasoline on a fire), and even the ability to block one specific bad query pattern at the ORM level. When a surge hits, you shed load surgically instead of falling over generally.

At this scale even altering a table is dangerous — changing a column type can force the database to rewrite every row. So the rules are strict: only lightweight changes, a hard 5-second timeout on any schema operation, indexes built concurrently (without locking the table), and no new tables, period. Backfills crawl along under rate limits and can take over a week. Slow and boring beats fast and down.