How do we avoid blowing up Prometheus with cardinality when we add context?

Decide upfront on 5–10 allowed labels (service, endpoint, version, region, shard, customer_tier). Block PRs that add unbounded labels (user_id, request_id). Use exemplars to carry high-cardinality context into traces without exploding metric series. Enforce budgets with promtool and lint in CI.

Do we need ML for predictive detection?

Not to start. Z-scores over rolling baselines, slope/derivative for queue age and lag, and correlation with change events catch 80% of pre-incident patterns. Once stable, you can try Kayenta-style canary analysis or simple anomaly detectors—but keep interpretability for on-call.

What about cost? Logs and traces aren’t cheap.

Sample spans (tail-based sampling for errors/long latency), keep logs structured and short, and push high-cardinality data to traces, not metrics. Use recording rules to precompute signals; query raw data only on demand. We often cut costs 20–40% while improving MTTR.

We have a legacy monolith. Will this still work?

Yes. Add OTel auto-instrumentation where possible, inject `trace_id` into logs, and ship a minimal set of metrics (requests, p95, errors, queue age). You can still gate rollouts (blue/green or canary) on those signals. GitPlumbers has done this for JBoss/WebLogic and older Spring stacks.

Reliability-observability · Dec 12, 2025 · 9 minute read

Stop Chasing Graphs: Build Correlation That Predicts Incidents (and Auto-Rolls Back)

Leading indicators, not vanity dashboards. Tie metrics, logs, and traces to triage and rollout automation so you fix the real issue—before PagerDuty wakes you.

Back to all posts

Stop Chasing Graphs: Build Correlation That Predicts Incidents (and Auto-Rolls Back)

Key takeaways

Implementation checklist