Loading...
Loading...
Client-side versus server-side registries, health checks, heartbeat and why DNS alone falls short
The last topic ended with five dependencies combining to roughly 99.5% for the caller and one trace ID, a request identifier passed along every hop, holding the story together. Now watch that story break before it starts. Checkout needs to call Payments. Yesterday Payments lived at one address; last night autoscaling killed that box and started two new ones with brand-new addresses. Checkout’s config file still points at the corpse. Requests fail, not because anything is broken, but because nobody told Checkout where Payments moved to.
Hardcoded addresses die the moment infrastructure moves. And it always moves: deploys, scaling events, crashes, new regions. Services need a living phone book that updates itself, that’s service discovery.
One sentence version: instead of memorizing where everyone lives, services ask a registry that always knows.
On startup it tells the registry: “I’m Payments, find me here.”
Miss enough health checks and you’re quietly erased. No funerals, no stale entries.
Checkout asks “where’s Payments?” and gets the current live list.
Either the caller chooses (client-side) or a balancer does (server-side).
Seen in: Eureka, the registry built for Netflix's cloud services, and Consul, HashiCorp's registry and health-check tool, in client mode
Seen in: cloud-provider load balancers and Kubernetes Services, the stable virtual address Kubernetes gives a group of pods
Discovery plus health checks plus a key-value store, across datacenters. The Swiss-army option when you outgrow basics.
Services get names, DNS, the internet's name-to-address lookup, resolves them, and the cluster's proxy routes. Zero setup if you’re already on Kubernetes, the container platform most teams run, and the reason many teams never buy anything else.
etcd is the strongly-consistent store, one where every reader agrees on the latest write, underneath Kubernetes itself. Eureka is the older registry built for Netflix's services, still common in Java Spring shops, the framework those teams build on.
Tempting, and sometimes enough. DNS already maps names to addresses. But DNS answers get cached everywhere, so a dead box lingers in caches for minutes. It carries no health signal, no metadata, no versions. Fine for humans finding websites. Too slow and too dumb for machines finding machines dozens of times a second. Registries exist because DNS answers the wrong question too slowly. A short TTL, the lifetime stamped on each DNS answer, does not fix it either: resolvers, operating-system caches, and client libraries routinely ignore short TTLs and hold answers for minutes, DNS carries no health or metadata signal, and every lookup adds resolution latency on the hot path. DNS answers where something was; a registry answers what is healthy right now, different questions, different freshness budgets.
| Need | DNS gives you | Registry gives you |
|---|---|---|
| Dead-box removal | Minutes (TTL expiry) | Seconds (health checks) |
| Extra info (version, zone) | None | Tags and metadata |
| Setup cost | Zero, already there | Real infrastructure to run |
Every registry is a freshness-versus-chatter trade with concrete defaults. Consul agents typically check health every 10 seconds and deregister after a few missed passes; Kubernetes kubelets, the per-machine agents that report container health, report every 10 seconds with a 40-second grace before pod eviction; Eureka clients heartbeat every 30 seconds and get evicted after 90. During the gap between death and deregistration, callers hold stale endpoints, at 1,000 requests per second with a 30-second detection window, that is up to 30,000 requests aimed partly at a corpse, saved only by client retries onto healthy instances.
| Knob | Sane starting value | Push it and… |
|---|---|---|
| Heartbeat / TTL | 5–10s heartbeat, 15–30s TTL | 1s heartbeats find death instantly and drown the registry in chatter at scale |
| Client cache of endpoints | 5–30s local cache + watch/long-poll, a standing query the registry answers when something changes | No cache means registry overload; minute-long caches route to ghosts |
| Retry across endpoints | Try next healthy instance once on connection refusal | Blind retry on every error amplifies partial outages into full ones |
Registries break in registry-shaped ways. A network partition splits the cluster into two halves that each believe the other half died, both sides deregister the “dead” services, callers get contradictory endpoint lists, and each half routes only within itself. Users see intermittent failures that correlate with nothing in any single service's dashboard. Strongly consistent stores, where every reader agrees on the latest write, like etcd refuse writes on the minority side (safe but partially unavailable); availability-first registries like Eureka keep serving possibly-stale lists on both sides (available but briefly lying).
The mundane killer is the mass re-registration storm: a whole availability zone, one data-center building in a region, blips, hundreds of instances re-register simultaneously, and the registry plus every watching client melts under the notification flood, the discovery layer becomes the outage's second victim. Jittered backoff on registration, randomly spread-out retries that avoid marching in lockstep, and watch batching exist for exactly this morning.
Register on boot, heartbeat every few seconds, deregister on missed checks, and let callers watch a cached list, Kubernetes DNS plus its built-in proxy is this pattern prebuilt, Consul is the buy-it-yourself version. Client-side discovery saves a hop and survives balancer outages but spreads lookup logic into every language; server-side keeps callers dumb at the cost of load-bearing middlemen.
Register on boot, heartbeat every few seconds, deregister on missed checks, and let callers watch a cached list, retrying once onto the next healthy instance when the 30-second detection window aims requests at a corpse. Found each other? Great. But finding is not understanding: whether Checkout points at a noun with a verb or calls a named function decides what can be cached, what can be retried, and what breaks when either side changes. Two philosophies have been arguing about it for twenty years: name the thing, or name the action?