Loading...
Loading...
Dynamically locate services in a distributed system
Client Routing, Automatic Registration & Health Checks
You already know from the DNS module how names resolve to addresses with caching delays, and from the load-balancing module how health probes decide which backends stay in rotation. Autoscaling killed two boxes, started three, every address changed. Checkout still calls the old ones and fails. In a world where machines are cattle, nobody memorizes addresses. A registry (a small always-on directory where services announce themselves and callers look them up) does: services announce themselves, callers ask it, the dead get pruned by health checks (repeated small probes that mark silent machines unhealthy).
Think of a living office directory, the one comparison we will use here: newcomers pin their room number on arrival, movers update their card, and the receptionist crosses out anyone who stops answering roll call. In the lab above, kill an instance and watch the directory cross it out within a few probe rounds while callers keep succeeding, because nobody dials from memory anymore.
The naive setup hardcodes numeric addresses into config files and redeploys to change them. We rejected freezing that list because that works for pets (long-lived servers you nurse) and fails for cattle (short-lived containers you replace hourly), where every scale event, crash, and deploy rewrites the truth. The registry moves the truth out of configs into one live lookup.
Registration (announcing name, address, and port on startup), health checking (proving liveness on a schedule), discovery (asking the desk for currently healthy addresses), and choosing (picking one, usually round-robin) run in that order. The ordering matters: choosing before checking sends users to corpses. In the lab, watch a fresh box sit out until its first passing probe, then join the rotation, because announcements earn trust only after roll call.
Client-side discovery (each caller queries the desk and balances across answers) keeps the path short: caller to desk to box. The price is coupling, meaning every language and framework you use must learn the desk's protocol plus caching plus retry. Use it when callers are few and sophisticated. In the lab, add a second caller language and feel the duplicated logic, because the desk did not get simpler, the callers multiplied.
Server-side discovery (callers dial a balancer, which queries the desk and forwards) keeps callers dumb: one address, zero directory code. The price is the middleman itself, another hop plus another thing to scale and survive. Use it when callers are many, small, or written by other teams. The edge case is the middleman's own cache: a 60-second entry means a dead box keeps receiving calls a full minute after the desk crossed it out.
Active probing (the desk calls each box's health endpoint every N seconds) catches hangs the box itself would never report, at the cost of probe traffic that grows with fleet size: 1,000 boxes every 10s means 100 probes per second of overhead. Heartbeats (each box posts I-am-alive on a schedule with an expiry TTL, meaning time to live) reverse the direction so the desk only tracks expiries, at the cost of trusting a dying box's last word. Most desks blend both: heartbeats for speed, probes for truth. In the lab, freeze a box without killing it and watch heartbeats lie while probes tell the truth.
The desk fetches a health endpoint every N seconds and pulls slow or failing boxes fast.
Each box renews its lease every 10s with a 30s expiry. Three missed renewals and the card drops.
Tools like Consul (a directory service with health checks and DNS answers that spans data centers) fit virtual machines and mixed fleets. You run the desk yourself and gain one directory for every language at once.
Kubernetes (a container platform where short-lived pods come and go) gives every stable service name a directory entry that tracks its current pods. Code dials the name and the platform keeps the addresses fresh, which is discovery hiding in plain sight.
Nobody memorizes addresses anymore, the directory knows, the roll calls prune, the traffic flows. But knowing where a service lives says nothing about how to talk to it well. When a downstream call hangs instead of failing outright, how long should you wait before giving up and trying the next address?
Try this in the playground
Open a template and build it yourself — then take a quiz.