How do I profile safely in Kubernetes without changing the app?

Use eBPF-based profilers like Parca or Pyroscope’s eBPF agent. They sample kernel/user stacks without in-process agents. Scope via namespace and limit overhead by sampling at 1–10 Hz during incidents.

Should we scale first or optimize first?

Scale first to stop the bleeding (HPA, cache, rate limit). Then optimize with profilers and EXPLAIN ANALYZE. Always separate mitigations (reversible, fast) from optimizations (risky, lasting).

What if I don’t have a prod-like staging for load tests?

Carve out a shadow env that mirrors prod topology with 10–20% of prod capacity. Replay sampled traffic with tc mirroring or service mesh traffic split. If that’s impossible, use a small canary with strict abort thresholds.

How do I make sure a change actually improved things?

Define success criteria upfront (e.g., p95 < 300ms, CPU < 70%). Run the same workload before/after, compare Prometheus metrics, and only complete rollout if both latency and error rate improve. Record in a runbook PR.

Any quick JVM GC win without deep tuning?

Switch to G1GC if you’re on CMS/Parallel, set Xms=Xmx to avoid catastrophic resizing, cap pause target ~200ms, and fix allocation hotspots first. Most wins come from reducing churn, not GC flags.

Are feature flags safe during performance work?

Yes—if they’re typed and server-evaluated with consistent hashing. Pair with canary analysis so you can roll back instantly when a flag increases error budget burn.

Guides · Oct 18, 2025 · 10 minute read

Five Battle‑Tested Performance Playbooks: CPU Hot Paths, DB Latency, GC Pauses, I/O Stall, and Lock Contention

Q: Any quick JVM GC win without deep tuning?

Switch to G1GC if you’re on CMS/Parallel, set Xms=Xmx to avoid catastrophic resizing, cap pause target ~200ms, and fix allocation hotspots first. Most wins come from reducing churn, not GC flags.

Q: Are feature flags safe during performance work?

Yes—if they’re typed and server-evaluated with consistent hashing. Pair with canary analysis so you can roll back instantly when a flag increases error budget burn.

When prod melts, you don’t need platitudes. You need playbooks with hard checkpoints, precise metrics, and the right tools. This is what we actually run at 2 a.m.

Back to all posts

Five Battle‑Tested Performance Playbooks: CPU Hot Paths, DB Latency, GC Pauses, I/O Stall, and Lock Contention

Key takeaways

Implementation checklist