Loading...
Loading...
Protecting services under failure — circuit breakers, bulkheads, and graceful degradation
The warehouse recovers by rerunning model 7 of 12 while dashboards keep reading stale-but-consistent marts. The serving path has no nightly rerun and no stale mart to hide behind. Service A calls B synchronously, which is waiting for the answer before doing anything else, and B hangs for 5 seconds. Every one of A threads blocks holding its caller connection, A queue fills, and soon callers of A time out too. Think of an apartment building wiring as the single analogy here: one shorted appliance should trip its own breaker and darken one room, not melt the riser and take the whole building. A cascade, which is exhaustion propagating upstream as each layer waits on the layer below, is the building without breakers.
The naive fix is waiting patiently with generous timeouts, on the theory that B recovers and every call eventually succeeds. That fails because patience consumes the scarcest resource, which is threads and connections held open per waiting call: at 200 requests per second with 5-second hangs, A needs 1,000 stuck threads, the pool dies in seconds, and C calling A dies next. Resilience patterns break this chain by hedging load and shedding work before the whole mesh collapses.
No pattern: A -> B (5s hang) -> A threads exhausted -> C calling A also hangs -> outage With breaker: A -> B failure rate 50% -> breaker OPEN -> A fails fast -> serves fallback/cached
A circuit breaker, which is a guard per dependency with closed, open, and half-open states, passes calls while closed, fails fast while open, and probes with limited traffic while half-open. It trips when the error rate passes a threshold inside a window, such as 50 percent over 10 seconds, then cools down for 30 seconds before allowing 5 probes.
A bulkhead, which is a partition of thread and connection pools per dependency so one slow downstream cannot steal every slot, gives 20 threads to payments and 20 to search rather than a shared 40. Compartment by compartment, one flood never sinks the ship.
Load shedding, which is deliberately refusing low-priority traffic to protect the core path, triggers at 95 percent CPU or queue past 1,000: drop background jobs with a 503, which is the HTTP status meaning service unavailable, plus Retry-After hint, which tells the caller when to return, while keeping user-facing requests. Enforce with token buckets at ingress and priority lanes at the balancer.
Service A waits 30 seconds on B while holding a thread per call. At 200 requests per second that is 6,000 stuck threads, because 200 times 30 equals 6,000 concurrent waits, and the pool dies in seconds with C calling A dying next. The entire cascade is missing deadlines: no per-attempt timeout, which is how long one try may take, no end-to-end deadline, which is the caller total budget propagated downstream, and no retry budget, which caps retries as a fraction of live traffic. Every call gets a deadline shorter than its caller patience, or patience runs out everywhere at once.
deadlines that survive review:
per-attempt timeout: p99 downstream + headroom (e.g. 800ms for 500ms p99)
p99, which is the latency only the slowest 1 percent of requests exceed
end-to-end deadline: propagate caller budget, never exceed it downstream
retries: max 2-3, ONLY idempotent GETs, backoff 100ms -> 400ms + jitter
retry budget: cap retries at 10-20% of live traffic (extra gets shed)
hedged requests (p99 hedge for read-only fan-out):
fan-out, which is sending one request to many replicas and taking the first answer
send 2nd request at p95 latency (e.g. 300ms), take first response
use ONLY when downstream has headroom — hedging a saturated service kills itB stumbles, and every caller retries instantly three times: arrival rate quadruples exactly when capacity halves, because each original request becomes one plus three retries. One team tried five retries with no budget instead of two with one: each request became six, so a 30 percent capacity dip arrived as a 6x flood and finished the outage the dip started. Exponential backoff with jitter, which spreads retries across time with random variation, plus a retry budget that drops retries past 20 percent of traffic, converts the storm into a drizzle.
Serving stale cache when B is down is correct, until the cache path shares B thread pool and blocks behind it. Isolate fallback execution on a separate pool with a short 50-millisecond cache timeout, or the lifeboat sinks with the ship it was meant to rescue.
Adding a breaker without thresholds fails quietly. A breaker that trips on one error flaps all day on noise, while one needing a thousand errors never trips before the cascade finishes. The window, which is how many calls or seconds form the sample, the failure rate, which is what fraction trips the breaker, the cooldown, which is how long it fails fast before probing, and the probe count are what turn the pattern from decoration into protection.
circuit breaker (per dependency, Resilience4j shape):
Resilience4j, which is a Java library packaging breakers, bulkheads, and limiters
slidingWindow: 100 calls or 10s
failureRateThreshold: 50% # open when half the window fails
slowCallThreshold: 1s, slowCallRateThreshold: 60%
waitDurationInOpenState: 30s # fail fast, serve fallback
permittedInHalfOpen: 5 probes # one success != healthy
minimumNumberOfCalls: 20 # don't judge on 3 calls
bulkhead (per dependency pools):
pool B (payments): 20 threads, queue 50, timeout 800ms
pool C (search): 20 threads, queue 50, timeout 500ms
NEVER shared 40 — slow B must not steal from C
load shed at ingress:
queue > 1000 or CPU > 85%: 503 + Retry-After on background tier firstBreakers, bulkheads, and shedding keep one failure from becoming every failure. But the breaker only knows the error rate crossed 50 percent over 10 seconds. It cannot say which deploy at 14:02 did it, or whether the fallback actually served. Firing blind is still blind.