Loading...
Loading...
Long polling, WebSockets, and Server-Sent Events for real-time APIs
The ledger rebuilds any past moment by folding events, but folding forty million events takes minutes. The chat window needs the next message in 200 milliseconds, not after the next rebuild. Polling, which is the client asking are we there yet on a fixed interval, every second wastes most requests on empty answers: fifty thousand users polling each second fires fifty thousand requests for roughly one thousand real messages, so 98 percent of the traffic buys nothing. Long polling, which is the server holding each request open until an event arrives or a timeout hits, ties up one server thread per waiting user, and at one megabyte of stack per thread ten thousand waiting users eat ten gigabytes of RAM before serving anyone. Think of a phone call held open against checking a mailbox as the single analogy here: polling walks to the mailbox repeatedly, long polling waits at the mailbox door, and true push keeps the call connected so either side speaks the moment they have something to say.
The naive fix is to poll faster, which trades freshness for load without ever reaching freshness: halving the interval doubles the requests while still missing anything that arrives between checks. The real solution is a persistent channel where the server pushes frames, which are small chunks sent over an open connection, without holding one thread per connection, and the choice is which kind of call to hold open.
| Pattern | Direction | Overhead | Use |
|---|---|---|---|
| Short poll | Client asks on an interval | High, mostly empty responses | Rare checks where seconds of delay are fine |
| Long poll | Server holds each request until an event or timeout | One held request per waiting client | Simple push without protocol upgrades |
| SSE | Server to client stream over plain HTTP, meaning events flow one way on a normal web request | One connection with text event frames and auto-retry | Feeds and notifications that flow one way |
| WebSocket | Both directions over one upgraded connection | One connection with binary and text frames | Chat, games, and collaborative editing |
SSE, which is server-sent events, a one-way stream of text events over a normal web request, starts as a GET to an events endpoint returning an event-stream content type. The browser EventSource, which is the built-in client for SSE, reconnects automatically and sends the last seen event id so the server resumes without duplicates. It inherits HTTP load balancing and multiplexing, which is sharing one connection among many streams, for free.
A WebSocket, which is a bidirectional channel opened by upgrading a web request to a persistent framed connection, starts with an upgrade handshake returning status 101, then exchanges framed messages in both directions. It needs heartbeats, which are periodic ping and pong frames proving the connection is alive, plus explicit close handling, and every proxy in the path must allow the upgrade headers through.
Your chat reaches 50,000 concurrent users. One application process holds roughly 10,000 to 30,000 idle WebSockets before memory and file descriptors, which are per-connection handles the operating system limits, start biting. At 50,000 that means at least 3 to 5 application servers plus a spare for deploys, and every server must learn about every message through publish-subscribe fan-out, which is publishing once so all servers with room members deliver locally. The arithmetic: 50,000 divided by 15,000 per node is about 4 nodes plus 1 spare, memory at 50,000 times roughly 50 kilobytes per socket is about 2.5 gigabytes just for buffers, and fan-out of 1 message times 200 room members is 200 deliveries, so 1,000 messages per second in busy rooms becomes 200,000 deliveries per second the bus must absorb.
capacity math: 50,000 conns / 15,000 per node ~= 4 nodes + 1 spare
memory: 50k x ~50 KB per socket ~= 2.5 GB just for buffers
fan-out: 1 chat msg x 200 room members = 200 deliveries
1,000 msgs/s in busy rooms = 200,000 deliveries/s — pub/sub must absorb this
latency budget: typing indicator < 300ms, chat < 200ms p99
heartbeat ping every 25s, dead after 2 missed pongs (~60s)
LB idle timeout must exceed heartbeat or proxies kill live sockets
LB, which is the load balancer distributing connections across edge serversSticky sessions, which pin each connection to the server that accepted it, mean rolling deploys or autoscale evictions sever thousands of sockets at once, and every client reconnects in a thundering herd. Terminate at the edge, keep session state in shared memory such as Redis rather than process memory, and let any server serve any reconnect.
A dashboard refreshing every 30 seconds across flaky mobile networks wastes less than 50,000 held sockets fighting network address timeouts. Use the persistent call for typing indicators and presence, which is who is online now, and plain fetching for history: match the protocol to the freshness need rather than the hype.
Everything passes on your laptop, then production messages stop arriving after 60 idle seconds. The corporate proxy and the cloud load balancer both closed the idle connection, because your heartbeat ran every 120 seconds while their timeout sat at 60. Realtime systems die on middleboxes, which are proxies and balancers between client and server, more than on code: upgrade headers stripped so the call never upgrades, idle timeouts shorter than pings, and newer multiplexed connections behaving nothing like your older test setup.
survival config:
ws heartbeat: ping every 25s, pong timeout 10s, close after 2 misses
LB idle timeout: 120s (above heartbeat interval)
proxy: allow "Upgrade: websocket" + "Connection: Upgrade" passthrough
SSE: retry: 3000 ms + Last-Event-ID resume; X-Accel-Buffering: no
reconnect: exponential 1s -> 30s max with jitter, resume from last idA user on server 1 sends to a room with members on servers 1 through 4. Server 1 publishes once to a shared channel on Redis or NATS, which are fast messaging buses used for fan-out between servers, and every server with a room member delivers locally to its own sockets. Presence rides the same bus as expiring keys: online status is just a heartbeat key with a 60-second expiry, refreshed every 25 seconds, so a vanished user times out into offline without any explicit logout.
publish: room:8821 -> {"msg_id":"m-1042","from":"ada","text":"hi"}
each edge server: SUB room:8821, deliver to local members only
dedup on msg_id: reconnect replays last 50 msgs, client drops seen ids
presence: SETEX presence:ada 60 "server-3" (refresh every 25s)One-way feed goes over server-sent events, real conversation goes over WebSocket, and fifty thousand sockets spread over four edge nodes through the bus above. Then traffic doubles and you add a fifth node: which rooms move, and how do you avoid every client reconnecting to a stranger that knows nothing? The answer starts with a remainder operation that moves eighty percent of everything.