Tracing That Survives Prod: A Pragmatic Playbook for Microservices (OpenTelemetry, Meshes, and Messy Reality)

The outage you’ve lived through

It’s 12:41 AM. Checkout is timing out. Grafana shows a red wall. Half the team is grepping logs, the other half is clicking random Jaeger traces that end at the gateway. You ship microservices, but your traces die at hop two and async hops are a black hole. I’ve seen this movie at unicorns and at 20-year-old enterprises. Tracing fails not because tools are bad, but because propagation is partial, sampling is naive, and nobody owns the collector.

This is the playbook we use at GitPlumbers to make tracing boring, reliable, and cheap enough to keep. It’s opinionated because the alternatives cost you MTTR and money.

Decide your standards and topology (don’t skip this)

Before you touch code, decide the rules of the road. If you let each team “pick what works for us,” you’ll be back here in six months.

• Propagation: Use W3C traceparent/tracestate end-to-end. Only translate B3 at legacy edges. Ban custom headers.
• Context semantics: Standardize service.name, service.version, and deployment.environment via OpenTelemetry Resource attributes.
• Backend: Pick one primary trace backend: Jaeger, Grafana Tempo, Honeycomb, or Datadog. All can ingest OTLP. Avoid dual-writes from apps; fan-out in the OpenTelemetry Collector.
• Topology: Run an otel-collector as a DaemonSet (per-node) or per-namespace. Apps export to local collector. Collectors forward to backends and apply sampling/policy. Mesh/ingress export to the same collector.

Rule: apps and proxies speak OTLP to a local collector. The collector speaks whatever to your vendors.

Checkpoints

• A one-pager in your repo: propagation = W3C, backend = X, collector address = otel-collector:4317, required resource attributes listed.
• A “no B3 past the edge” admission policy if you’re on Kubernetes.
• A runbook for changing sampling without redeploying apps.

Instrument the edges first, then services, then async

Tracing dies at boundaries. Instrument those first or your beautiful service spans won’t correlate.

1. Ingress/gateway

   • NGINX Ingress: enable tracing with an OpenTelemetry module or sidecar exporter. For vanilla ingress-nginx, forward headers and let app SDKs create spans; or front with an Envoy gateway.
   • Istio (1.18+): use Telemetry API to send spans to the collector.

   apiVersion: telemetry.istio.io/v1alpha1
   kind: Telemetry
   metadata:
     name: mesh-tracing
     namespace: istio-system
   spec:
     selector: {}
     tracing:
     - providers:
       - name: otel
       randomSamplingPercentage: 10
     providers:
     - name: otel
       otel:
         service: otel-collector.otel.svc.cluster.local
         port: 4317

   • Checkpoint: Incoming requests show a root span at the gateway with a trace_id you can follow into at least one downstream service.

2. Service auto-instrumentation

   • Node.js (Express)

     // tracing.js
     const { Resource } = require('@opentelemetry/resources');
     const { SemanticResourceAttributes: S } = require('@opentelemetry/semantic-conventions');
     const { NodeTracerProvider } = require('@opentelemetry/sdk-trace-node');
     const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-grpc');
     const { BatchSpanProcessor } = require('@opentelemetry/sdk-trace-base');
     const { registerInstrumentations } = require('@opentelemetry/instrumentation');
     const { HttpInstrumentation } = require('@opentelemetry/instrumentation-http');
     const { ExpressInstrumentation } = require('@opentelemetry/instrumentation-express');
     const { propagation } = require('@opentelemetry/api');
     
     const provider = new NodeTracerProvider({
       resource: new Resource({
         [S.SERVICE_NAME]: 'checkout',
         [S.SERVICE_VERSION]: process.env.VERSION || '1.2.3',
         'deployment.environment': process.env.ENV || 'prod'
       })
     });
     
     provider.addSpanProcessor(new BatchSpanProcessor(new OTLPTraceExporter({
       url: 'http://otel-collector:4317'
     })));
     provider.register();
     
     registerInstrumentations({
       instrumentations: [new HttpInstrumentation(), new ExpressInstrumentation()]
     });

   • Java (Spring Boot 3+)

     # build.gradle
     implementation 'io.opentelemetry:opentelemetry-bom:1.36.0'
     runtimeOnly 'io.opentelemetry.javaagent:opentelemetry-javaagent:1.36.0'

     # JVM flags
     -javaagent:/otel/opentelemetry-javaagent.jar \
     -Dotel.service.name=payments \
     -Dotel.exporter.otlp.endpoint=http://otel-collector:4317 \
     -Dotel.propagators=tracecontext,baggage

   • Go (gRPC client/server): use go.opentelemetry.io/otel, add unary interceptors for client/server.

   • Checkpoint: For one request, you see a trace spanning gateway → service A → service B, with latency and status in each span.

3. Async boundaries (Kafka/RabbitMQ/SQS)

   • Inject/extract context in message headers. With OpenTelemetry JS:

     const { context, propagation } = require('@opentelemetry/api');
     const headers = {};
     propagation.inject(context.active(), headers); // producer
     producer.send({ topic: 'orders', messages: [{ key: id, value: body, headers }] });
     
     // consumer
     const ctx = propagation.extract(context.active(), message.headers);
     context.with(ctx, () => handleMessage(message));

   • For Java, use io.opentelemetry.instrumentation:kafka-clients and ensure header propagation is enabled.

   • Checkpoint: A trace shows producer → broker (as annotation) → consumer spans, not two disjoint traces.

4. Downstream calls (DB/HTTP)

   • Enable client instrumentation for pg, mysql2, redis, http, aws-sdk, etc. For Spring, it’s auto with the Java agent.
   • Checkpoint: Spans include db.system, db.statement (sanitized), http.url, http.status_code attributes.

5. Error capture

   • Record exceptions and set status.error on spans. Ensure your frameworks aren’t swallowing exceptions before the tracer sees them.
   • Checkpoint: Error traces highlight red spans; sampling boosts error traces (see next section).

Wire the collector and the mesh (your policy brain)

The OpenTelemetry Collector is where you do smart routing, sampling, and vendor fan-out. Treat it like an SRE-managed system with config in Git.

• Kubernetes deploy: run per-node for resilience.

apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-collector-config
  namespace: otel
data:
  otelcol.yaml: |
    receivers:
      otlp:
        protocols:
          grpc:
          http:
    processors:
      batch:
      attributes/redact:
        actions:
          - key: user.email
            action: delete
      tail_sampling:
        decision_wait: 5s
        policies:
          - type: status_code
            status_code:
              status_codes: [ERROR]
          - type: latency
            latency:
              threshold_ms: 500
          - type: probabilistic
            probabilistic:
              sampling_percentage: 10
    exporters:
      otlp/honeycomb:
        endpoint: api.honeycomb.io:443
        headers: { 'x-honeycomb-team': '${HONEYCOMB_KEY}' }
      jaeger:
        endpoint: jaeger-collector.jaeger:14250
        tls: { insecure: false }
      logging:
        loglevel: warn
    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [attributes/redact, tail_sampling, batch]
          exporters: [jaeger, otlp/honeycomb]

• Istio/Envoy: confirm traceparent is preserved and sampling at the mesh aligns with collector policies. If the mesh does 10% head sampling and collector tail-samples, you’ll only ever see 10%.
• Ingress-NGINX: ensure proxy_set_header traceparent $http_traceparent; and disable any header rewrites.
• Checkpoint: A load test shows stable ingest at the collector with backpressure and retries; no app writes directly to vendors.

Make traces useful: correlate logs and metrics, not just pretty graphs

You’re not buying wall art. Traces must connect to logs and metrics.

• Log correlation

  • Ensure every log event includes trace_id and span_id.

  • Example (JSON logs):

    {"level":"error","msg":"charge failed","trace_id":"4bf92f3577b34da6a3ce929d0e0e4736","span_id":"00f067aa0ba902b7","order_id":"123"}

  • For Java agent: -Dotel.instrumentation.log-logging.enabled=true. For Node, add a pino/winston hook to pull IDs from @opentelemetry/api.

• Prometheus exemplars

  • Expose histograms with exemplars so you can click from a p99 latency spike to a representative trace in Grafana/Tempo.

  • Go example:

    httpRequestDuration.WithLabelValues("/checkout").ObserveWithExemplar(1.2, prometheus.Labels{
      "trace_id": trace.SpanFromContext(ctx).SpanContext().TraceID().String(),
    })

• Dashboards that matter

  • RED for each service: Rate, Errors, Duration, with trace-exemplar links.
  • A “Hot Paths” dashboard: top endpoints by latency contribution; click-through to traces.
  • Checkpoint: From any SLO burn alert, you can pivot to a trace in ≤2 clicks, see the slow hop, and the owner on-call.

Sampling, storage, and cost control (the part finance will love you for)

I’ve watched teams nuke budgets by tracing 100% of traffic. Don’t. Be deliberate.

• Start with head sampling at the edge: 5–10% is plenty for healthy traffic. Errors should be 100%.
• Add tail-based sampling in the collector: keep slow or error traces regardless of head decision. Use policies like status code, latency, or attribute match (e.g., customer.tier = enterprise).
• Dynamic sampling: elevate sampling on active incidents or for canary releases.
• Retention: 7–14 days hot, 30–90 days cold (Tempo object storage is cheap). Jaeger + Elasticsearch will bite you; prefer Tempo/Cortex/Thanos-style backends for scale.
• Cardinality discipline: avoid unbounded span.setAttribute with IDs. Cap baggage to <1KB total. Drop PII at the collector.
• Overhead targets: tracing adds <5ms p95 per request, <2% CPU overhead. Measure it in load tests.

Checkpoint metrics

• Trace coverage: >90% of requests generate a trace (before sampling), measured by requests vs spans created.
• Propagation success: >99% of traces have at least 2 services linked (not orphaned spans).
• Error capture: >95% of 5xx requests have an error span.
• Cost: storage/ingest per request within budget (track $/K requests).

A 30‑day rollout plan with proof points

Week 1 – Standards and collector

1. Lock propagation (W3C), pick backend, deploy otel-collector with config in Git (GitOps via ArgoCD).
2. Instrument ingress/gateway; verify root spans. Set 10% head sampling.
3. Prove: 1 service and gateway appear in one trace under load; collector handles 2x peak QPS.

Week 2 – Core services

1. Add auto-instrumentation to top 3 traffic services (Java agent, Node SDKs, Go interceptors).
2. Enable DB/HTTP client spans; scrub SQL text at collector if needed.
3. Prove: gateway → svc A → svc B traces with attributes and 0.99 propagation.

Week 3 – Async and correlation

1. Instrument Kafka/RabbitMQ producers/consumers for header propagation; backfill libraries if missing.
2. Add log correlation and exemplars.
3. Prove: a trace travels across a message bus; Grafana panel links to a trace from p99.

Week 4 – Sampling polish and guardrails

1. Turn on tail-based sampling for errors and >500ms latency, reduce head sampling to 5%.
2. Add chaos tests: drop traceparent at random at the gateway; ensure SDKs create new roots but still sample errors.
3. Prove: MTTR drills—on injected 500s, on-call can isolate the slow hop in <10 minutes.

Exit criteria

• Coverage >90%, propagation >99%, overhead <5ms p95, error traces sampled at ~100%.
• Dashboards exist for RED + hot paths; runbook documented. Budgets approved with measured ingest/storage.

What breaks in the wild (and how we’ve fixed it)

• Mesh vs SDK sampling fights: Istio sampled 1%, collector tail-sampled for errors; healthy error traces were lost. Fix: raise mesh sampling or move all sampling to the collector via x-ot-span-context honoring—better, keep mesh at ≥10% when tail sampling is on.
• Async orphans: Kafka libs not carrying headers across frameworks (seen in older Spring Cloud Stream). Fix: standardize on OpenTelemetry instrumentation version and add a thin wrapper to inject/extract; add CI tests that assert header presence.
• Hot attribute cardinality: teams added user_id as an attribute. Tempo storage exploded. Fix: hash IDs if needed, otherwise log it—not in spans. Enforce via collector attribute processor.
• PII in spans: ORM auto-captured SQL with literals. Fix: parameterized queries, and a collector attributes/delete list for known fields.
• Vendor lock whiplash: app SDKs exported to Datadog directly; switching to Honeycomb was a month-long migration. Fix: all apps export OTLP to collector; collector fans out.
• CI regressions: a refactor dropped context in an async callback chain. Fix: unit tests that assert trace_id continuity; chaos tests that drop headers and verify graceful re-rooting.

If any of this sounds familiar, you’re not alone. Tracing isn’t a feature; it’s a contract. Treat it that way.

Quick reference: what “good” looks like

• Standards: W3C traceparent, OTLP to collector, resource attributes set.
• SDKs: Java agent 1.36+, Node OTel SDK 0.50+, Go OTel 1.22+.
• Backends: Tempo for scale/cost, Jaeger for UI/primitives, Honeycomb for high-cardinality analysis, Datadog for integrated infra.
• Policies: Head 5–10% + tail for errors/slow, redact PII in collector, drop high-cardinality attributes.
• Dashboards: RED + exemplars, Top N slow endpoints, Error heatmap by service.
• SLOs: trace coverage, propagation success, overhead, cost per request.
• Ownership: SRE owns collector and sampling. App teams own SDKs and context at boundaries.

When you hit those, you’ll stop asking “which service is slow?” and start fixing it in minutes.