Should I pick Loki or Elasticsearch for hot search?

If most of your queries are recent and label-filterable, Loki is cheaper and simpler to scale. If you rely on deep full-text search across arbitrary fields, Elasticsearch/OpenSearch shines — but control cardinality and use ILM. Many teams run Loki hot + S3 cold and keep ES only for specific teams that need text search.

Do I need OpenTelemetry for logs?

You need consistent correlation. OTel makes trace propagation and log linkage boring. Even if you don’t ship OTLP logs, use OTel APIs to fetch trace/span IDs and include them in your logs.

How do I prevent sensitive data from leaking into logs?

Redact at the collector/agent with tested regex/transform processors. Add CI tests with known fixtures. Lock down cold storage access. Avoid logging raw headers/request bodies unless sanitized.

What’s a reasonable logging budget?

Target <3% of infra spend or <1.5 KB/request average for application logs. Sample debug logs and drop chatty, low-value events from hot tiers; retain full fidelity in cold storage if you must.

How do I measure if logging improved MTTR?

Run controlled incident drills before/after rollout. Track mean time from alert to root cause and the number of context switches (tabs/queries). Also measure query P95 and ingestion lag; they correlate strongly with triage speed.

Guides · Oct 28, 2025 · 10 minute read

The Logging Playbook I Wish We’d Had Before That 3 a.m. Outage

Q: What’s a reasonable logging budget?

Target <3% of infra spend or <1.5 KB/request average for application logs. Sample debug logs and drop chatty, low-value events from hot tiers; retain full fidelity in cold storage if you must.

Q: How do I measure if logging improved MTTR?

Run controlled incident drills before/after rollout. Track mean time from alert to root cause and the number of context switches (tabs/queries). Also measure query P95 and ingestion lag; they correlate strongly with triage speed.

A battle-tested, step-by-step guide to logs that actually help you debug — with schema, sampling, routing, retention, and correlation that won’t melt your budget.

Back to all posts

The Logging Playbook I Wish We’d Had Before That 3 a.m. Outage

Key takeaways

Implementation checklist