How do I add `trace_id` if I’m not using distributed tracing yet?

Start by generating a `request_id` at your ingress (NGINX/Envoy) and propagate it via headers. Include it in logs across services. When you add OpenTelemetry later, map `request_id` to `trace_id` during transformation so your saved queries keep working.

Loki or Elasticsearch?

If your primary use is K8s service logs with high volume and you can live with label-based filtering, Loki is cheaper and simpler. If you need ad-hoc field queries across arbitrary JSON and heavy aggregations, Elasticsearch/OpenSearch wins. Many shops run Loki for app logs and Elastic for security/audit.

How do I keep costs from exploding?

Drop DEBUG in prod at the router, sample high-frequency info logs (1–10%), enforce retention tiers (hot 48h, warm 7–14d, cold 30–90d), and watch cardinality (no per-user labels). Set budget alerts on ingest and storage, and review them weekly.

What about PII and compliance?

Redact at the edge (Vector/OTel). Maintain a denylist of fields (`password`, `ssn`, `card_number`) and run automated scans on downstream stores. Treat logging configs as code with security review. For regulated data, route to a separate, access-controlled index/tenant.

Do I need OpenTelemetry to get value?

No, but it multiplies the value. Even without OTel, structured logs with a propagated `request_id` and consistent schema will cut MTTR. Adding OTel later lets you pivot from trace to logs instantly, which is where the real speedup happens.

Guides · Oct 4, 2025 · 10 minute read

Your Logs Are Chatty, Not Helpful: A Field Guide to Debuggable Logging That Cuts MTTR in Half

If your on-call playbook starts with “open Kibana and pray,” this is for you. Here’s the logging strategy we’ve used to turn noisy firehoses into surgical tools — with concrete configs, guardrails, and rollout steps that survive real incidents.

Back to all posts

Your Logs Are Chatty, Not Helpful: A Field Guide to Debuggable Logging That Cuts MTTR in Half

Key takeaways

Implementation checklist