Debugging Distributed Systems: Logs, Metrics & Traces Visualized

Why is the API slow even though the server itself looks fast? The process may spend most of its time waiting on a database, cache, queue, identity provider, or third-party API. Distributed observability connects those boundaries.

The values in the explorer are illustrative rather than benchmarks. Its compact traces omit network clock skew, asynchronous links, batching, dropped telemetry, and vendor-specific conventions.

Three signals answer different questions

Logs record events: what happened at a particular moment? Metrics aggregate measurements: how often, how much, and how is that changing? Traces model one request as causally related spans: where did it go and where was time spent?

No signal replaces the others. A latency alert points to a time window, a trace identifies a slow dependency, and a correlated structured log can expose the safe diagnostic context for the failing span.

Structured logs

Machine-queryable fields beat sentences assembled for one anticipated search.

logger.info("order confirmed", {
  requestId,
  traceId,
  userId: safeUserReference,
  orderId,
  durationMs,
  outcome: "success",
});

Use levels consistently: debug for high-volume development detail, info for normal lifecycle events worth retaining, warn for recoverable abnormal conditions, and error for failures requiring investigation. Level names are policy, not universal truth.

Never log bearer tokens, passwords, full payment data, notification payload secrets, or unbounded request bodies. Redaction is part of instrumentation design.

Request and correlation IDs

A request ID identifies one ingress request. A trace ID links spans across services. A correlation ID can represent a broader business operation spanning requests and background jobs. Teams use these terms differently, so define them in the system contract.

Accept externally supplied identifiers only under a clear trust policy. Generate canonical internal IDs, return a safe request ID to clients, and propagate trace context using supported headers and instrumentation. HTTP Request Lifecycle shows every hop that must preserve it.

Metrics: counters, gauges, histograms

A counter increases: requests, errors, processed jobs. A gauge moves up and down: queue depth, active connections, memory in use. A histogram groups observations such as latency or payload size into buckets, enabling distributions and percentiles.

Metric labels create a time series per combination. Method and normalized route are usually bounded; raw user IDs, request IDs, and URLs can create unbounded cardinality and operational cost. Put high-cardinality identity in sampled traces or controlled logs instead.

Percentiles, not only averages

If 99 requests finish in 50 ms and one takes 5 seconds, the average is about 99.5 ms. It describes nobody's experience particularly well.

• P50 is the median: half the observations are at or below it.
• P95 leaves roughly five percent slower.
• P99 exposes a thinner tail where timeouts and resource contention often appear.

Percentiles need a population and time window. A P99 aggregated from precomputed service percentiles is generally not the true global P99; combine compatible histogram distributions or raw observations. Low traffic also makes high percentiles noisy.

Traces and spans

A trace contains spans representing timed operations. Parent/child relationships show causality and nesting. A span records a service or operation name, start and end, status, attributes, events, and context identifiers.

trace: checkout
  client request
    API handler
      authentication
      cache lookup
      database query

The waterfall bar shows wall-clock duration, not necessarily CPU consumption. Parent spans can overlap children, parallel children can overlap each other, and the critical path—not the sum of every duration—determines total latency.

Propagation makes a trace distributed

Instrumentation injects trace context into an outgoing request and extracts it in the next service. Missing propagation creates disconnected traces. Async queues need message context or explicit span links because job processing may happen much later and is not always a simple nested child.

The Background Jobs tutorial shows why one logical job may have multiple delivery attempts. Each attempt deserves visibility without pretending duplicates are separate business operations.

Timeouts, exceptions, and retries

Set timeouts from end-to-end budgets. If a mobile request allows two seconds, giving three nested dependencies two seconds each cannot meet it. Propagate cancellation where supported and distinguish deadline exhaustion from an explicit business rejection.

Record errors on the span that owns them and map status carefully. An HTTP 404 may be a normal product outcome; a recovered retry may leave the root request successful while one child attempt contains a 503. Hiding the failed attempt makes retry-driven latency and load invisible.

Sampling

Recording every trace can be expensive. Head sampling decides near the start and is simple but may miss rare failures. Tail sampling decides after more of the trace is known and can retain slow or failed requests, but requires collection infrastructure and buffering.

Sampling changes counts. Do not calculate business totals from a sampled trace set unless weighting and sampling policy make that valid. Metrics remain the usual source for complete aggregate rates.

Alerts and dashboards

Alert on symptoms users experience—error rate, latency objectives, stale queues, and failed critical flows—then use resource metrics for diagnosis. CPU at 80% is not automatically an incident; a healthy system may intentionally use available capacity.

A useful backend dashboard pairs throughput, errors, and latency with saturation of constrained dependencies. Split by bounded route or service dimensions. Annotate deployments so investigators can connect a change in behavior to a release.

A practical investigation

1. Confirm the symptom and affected population from metrics.
2. Select a slow or failed trace from the same window.
3. Follow the critical path and inspect error spans.
4. Query correlated logs by trace or request ID.
5. Compare with dependency, saturation, deployment, and retry metrics.
6. Change one cause, then confirm the user-facing distribution recovered.

For database-heavy traces, indexes and transaction waits are common next investigations. For client delay, separate network time from React Native rendering work.

Common failure cases

• Logging errors without request context produces isolated stories.
• Tagging metrics with user or request IDs creates cardinality explosions.
• Reporting only averages hides the slow tail.
• Marking every non-2xx response as an infrastructure error creates noise.
• Retrying without spans hides load amplification.
• Sampling without documenting policy makes dashboards misleading.
• Telemetry that contains secrets turns debugging data into a security incident.

Key takeaways

• Logs explain events, metrics explain distributions and trends, and traces explain request paths.
• Context propagation is what connects services.
• Percentiles reveal tails that averages conceal.
• A span measures elapsed operation time, not necessarily CPU work.
• Sampling controls cost but changes what can be concluded.
• Observability should shorten a real investigation, not merely produce more data.