Observability that answers a question at 3am

Pick the signal for the question

The design-round version is short: emit structured logs with a request id, record latency histograms per endpoint and dependency, and propagate a trace id at the edge so every hop can be stitched together. That is three decisions and it covers almost every incident.

SLOs and error budgets

An SLO is a target for a user-visible measurement: "99.9% of checkout requests succeed within 500 ms over 28 days". The remaining 0.1% is the error budget. If you burn it slowly, ship features; if the burn rate is high, stop shipping and fix reliability. This converts an argument about priorities into arithmetic.

Tracing in one paragraph

The gateway generates a trace id and passes it in a header. Each service creates a span with a parent, records start and end, and attaches the id to its logs. A slow request then shows a waterfall: 40 ms gateway, 15 ms auth, 1,900 ms inventory, 30 ms database. Without it, the same investigation is a sequence of guesses across five log files.

trace 7f3a
  gateway       40 ms
    auth        15 ms
    cart       610 ms
      redis     2 ms
      inventory 590 ms   <-- the slow hop
    payment    120 ms

Cardinality is a budget too

• Bound label values. Method, status class, endpoint template — not raw paths, not user ids, not order ids.
• Bucket latency. Histograms with fixed buckets; storing raw durations per request is a log, not a metric.
• Sample traces. 1-10% head sampling is enough for latency shape; keep all traces for errors.
• Log in structured form. One JSON object per event with level, request id, route and outcome. Unstructured strings force regex archaeology.

One cardinality trap worth naming: a /user/{id} endpoint recorded with the raw path creates one series per user. The fix is route templating at the instrumentation layer, and it is a genuine interview point because it is the difference between a working dashboard and a monitoring bill nobody approved.