How do I pick SLO thresholds without over-optimizing?

Start from user journeys and business outcomes. If users tolerate 1s on search but only 500ms on checkout, encode that. Use historical p95/p99 plus error budget math to set realistic targets. Revisit quarterly as traffic and features change.

Is profiling in production safe?

Yes, with care. Go pprof CPU at 60s sampling and JFR/async-profiler are low overhead when used briefly. Keep endpoints locked down (localhost/mtls), timebox profiles, and store artifacts off-box.

Why p99 and not average?

Averages hide pain. Users experience tail latency. p95 reveals systemic issues; p99 exposes outliers and contention that torpedo conversion. Track all three, alert on p99 for critical paths.

When should I cache vs. scale?

Cache if the data is read-heavy and tolerates staleness for a short TTL; scale if data is highly dynamic and correctness is critical. Often it’s both: microcache at the edge plus origin capacity and backpressure.

What about microservices and cross-service latency?

Invest in tracing. Add per-hop budgets and circuit breakers (Istio/Envoy). Use bulkheads so one slow dependency doesn’t cascade. A good playbook includes dependency maps and fallback behavior.

Guides · Nov 11, 2025 · 10 minute read

The Bottleneck Playbooks I Reach For When Prod Starts Smoking

Stop guessing. Ship a small set of rehearsed performance plays you can run at 3 a.m.—with checkpoints, metrics, and tools that actually work.

Back to all posts

The Bottleneck Playbooks I Reach For When Prod Starts Smoking

Key takeaways

Implementation checklist