Loading...
Loading...
The three pillars of observability — what to instrument, how to correlate, and why it matters
The breaker tripped at 50 percent errors over 10 seconds and served stale cache. Which half failed, and did the fallback actually serve? Users report slow checkout. Is it the database, the cache, or a struggling region? Without signals you guess, and every team guesses a different culprit. Think of a doctor visit as the single analogy here: temperature charts over time, lab notes per incident, and imaging that follows one patient through the building. Observability, which is the ability to ask arbitrary questions about the system without shipping new code, works the same way by correlating three signals: metrics, which are aggregated numbers over time, logs, which are discrete event records, and traces, which are per-request trees showing where time went.
The naive fix is adding a dashboard per service and calling it done. That fails because dashboards answer only the questions someone predicted: the novel failure needs the unpredicted question, such as which 2 percent of checkouts with which payment method slowed after which deploy. The real solution is correlated signals with shared identifiers, so one alert walks to one slow request in seconds.
| Pillar | What | Cardinality | Use |
|---|---|---|---|
| Metrics | Counters, gauges, and histograms, which are counts, current values, and bucketed distributions, aggregated over time | Low per label, cheap to store for months | Alerts, service objectives, trends |
| Logs | Discrete events as structured lines, preferably JSON with named fields | High, one line per request or event | Ground truth for the exact failing case |
| Traces | Span tree per request, where each span is one timed step from gateway to service to database | High, controlled by sampling | Latency breakdown, critical path, dependency graph |
Propagate a trace id, which is a unique identifier for one request across services, plus a span id per step, through headers following the W3C traceparent standard, log both as fields on every line, and expose them as metric exemplars, which are sample trace links attached to chart buckets. Then one alert walks metric to exemplar to trace to log without guessing.
Slow checkout arrives as a ticket with no numbers. The engineer who opens a RED dashboard, which shows request rate, error ratio, and duration per endpoint, sees checkout errors jump from 0.1 percent to 8 percent at 14:02 while the slowest percentile triples, then drops to the USE view, which shows utilization and saturation per dependency, and finds the database disk at 100 percent saturation. Two dashboards, ninety seconds, root cause named. Without them the same ticket becomes a four-hour guessing war between teams, because each team brings its own chart with no shared identifier.
RED per service/endpoint (what users feel):
Rate: req/s per route (checkout POST: 400/s baseline)
Errors: 5xx ratio (alert: > 1% for 5 min, page: > 5% for 2 min)
Duration: p50/p95/p99 histogram (alert: p99 > 800ms for 10 min)
USE per box/dependency (why it feels that way):
Utilization: CPU > 75% sustained, connection pool > 80%
Saturation: queue depth, disk wait, GC pause time
GC, which is garbage collection, the runtime reclaiming memory in stop-the-world pauses
Errors: DB slow-query count, downstream timeout ratio
SLO shape: 99.9% of checkout < 500ms over 30d -> error budget 43 min/mo
SLO, which is a service-level objective, the promised threshold the budget measuresAverage latency hides the 5 percent of users timing out behind a healthy mean. Export histogram buckets at 50ms, 100ms, 250ms, 500ms, 1s, and 2.5s so the slowest percentiles compute server-side, and attach exemplars linking slow buckets to sample trace ids for one-click diagnosis.
One JSON line per request with timestamp, INFO level in production, trace id, user id, route template, status, and duration in milliseconds. DEBUG stays in development, because in production it costs terabytes and leaks personal data into a system everyone can read.
Someone adds user id as a metric label on the request counter. With 2 million users times 50 routes, that is 100 million series, where a series is one unique label combination stored and indexed, and the metrics backend falls over with queries timing out exactly when the incident starts. High-cardinality identifiers, which are values with millions of distinct possibilities such as user ids and request ids, belong in traces and logs, never in metric labels. The edge case teams miss is the raw URL path: templated routes such as orders by id produce 50 series, while raw paths with one entry per order id produce millions.
label discipline:
OK labels: service, route template (/orders/:id), status class (5xx), region
NEVER labels: user_id, email, request_id, trace_id, raw URL path
series estimate: routes(50) x status(5) x region(4) = 1,000 series (fine)
with user_id(2M): 2M x 50 x 5 x 4 = 2B series (outage + invoice)
sampling that keeps the signal:
traces: 100% of errors + >p95 slow, 1% of success (adaptive by rate)
logs: INFO sampled 100% for errors, WARN+ always; DEBUG off in prodMetrics scream, logs confess, traces draw the map. With all three correlated, slow checkout stops being a mystery and becomes a row in a table with a name on it. One question remains, and no chart answers it: the budget says 43 minutes a month, and March spent 40 by week two. Do you freeze ships or spend the last 3 minutes on the risky migration?