Loading...
Loading...
Prevent cascading failures by failing fast when services are down
Prevents cascading failures. If the service fails too often, the breaker TRIPS (Opens), failing fast without waiting for timeouts. It then self-heals by testing traffic later.
You already know from the retries-and-timeouts module how bounded waits plus capped retries still pile threads onto a dead dependency, and from the microservices module how one slow hop fails an entire end-to-end path. Payments goes dark. Checkout keeps calling it, thousands of threads (workers waiting on a reply) hanging, timing out, retrying, until checkout dies too, then the app, then the evening. One corpse, three funerals. A circuit breaker, a wrapper around each outside call that counts failures and refuses to call for a while once they pile up, watches the failures pile up and then does the brave thing: stops calling entirely until the dead show signs of life.
Think of the breaker box in your hallway, the one comparison we will use here: when one circuit shorts, the switch trips and saves the house instead of letting every socket melt. In the lab above, let the downstream fail and watch callers pile up with the breaker off, then turn it on and watch the same failure become instant rejections while the caller stays healthy, because refusing fast beats waiting slow.
The naive code retries a failing call with no memory: every request waits the full timeout, holds its thread, and tries again. We rejected endless retry because the pileup math leaves no escape. If 200 requests per second each wait 3 seconds on a dead service, you hold 200 × 3 = 600 threads at once, and a server with 500 threads is already dead. The breaker refuses instead of waiting, so the same 200 requests cost nearly nothing.
Breakers live in three states. Closed means current flows and calls pass through while the breaker counts. Open means the switch tripped and calls reject instantly. Half-open means a cautious test after a rest: let a few calls through, and close on success or trip again on failure. Try each transition in the lab by raising the failure rate, then waiting out the timeout, then letting the probes succeed and fail.
Closed means requests flow normally while the breaker tracks the failure rate over a window, for example 5 failures in 10 seconds. Occasional blips pass through because the count stays under the line. The edge case is a slow leak: 4 failures every 10 seconds never trips a count of 5, yet users still suffer, which is why many breakers also trip on percentage, like 50% failing over 20 calls.
Requests pass through normally while the breaker tracks success and failure counts.
Open means every call rejects immediately with a breaker-open error instead of touching the downstream. That fail-fast reply (an instant error rather than a slow timeout) is what saves the threads. After the rest timeout, say 60 seconds, the breaker does not jump straight to closed. It moves to half-open for a careful test, because slamming a sick service with full traffic just trips it again.
Half-open means a small probe budget, say 3 test requests needing 2 successes, flows through while the rest still reject. Success closes the breaker back to normal. Any probe failure trips it open again for another full rest. The nuance is sizing the probe budget: one probe flaps on a single lucky success, while fifty probes is just the old flood wearing a disguise. In the lab, set probes to 1 versus 5 under a flaky downstream and watch the first flap while the second holds steady.
Four settings shape every breaker. Set them from your latency budget, not from defaults. Too eager and normal blips trip the house. Too patient and the threads drown before the switch moves.
Trips after this many failures in the window. Lower trips faster but flaps on blips. Start from error budget: if 1% of 500 calls per 10s may fail, 5 is the line.
How long to reject before probing. Longer rests a sick service better but delays recovery. Match it to restart time: 30 to 60 seconds for a cold service.
Successes in half-open before trusting the service again. Two of three tolerates one flaky probe without declaring victory on one lucky call.
Longest wait for one downstream reply before counting a failure. Set it near the 99th percentile latency plus slack: if normal is 200ms, 1000ms marks truly stuck.
An open breaker must still answer something. Pick the fallback per endpoint before the outage, because inventing one mid-incident is how empty checkouts ship.
Serve the most recent success with a stale warning. Right for prices and feeds, wrong for money movement.
Return an empty list or default settings. Safe only where empty is obviously not success.
Keep the core path working and disable the garnish, like checkout without recommendations. Say what is missing in the reply.
Route to a second service for the same job. Powerful and double the bills and failure modes, so reserve it for payments and messages.
The first widely copied breaker library, a Java toolkit built inside a large streaming company to quarantine failing dependencies, proved the pattern: wrap the call, count, trip, probe, recover. Its dashboards made the three states visible to whole teams.
Later libraries, small functional toolkits with no dependencies used by Java service frameworks, keep the same closed, open, and half-open loop with less machinery. The states matter more than the brand, so learn the loop once and read any of them.
Stop calling the dead, fail fast, try again later, three positions of one switch. Breakers do not prevent failures, they quarantine them, which is the difference between one bad dependency and one bad evening. For your own checkout path, which endpoints deserve a stale-cache fallback and which ones must visibly fail rather than risk a wrong answer?
Try this in the playground
Open a template and build it yourself — then take a quiz.