The Performance Playbook That Keeps You Out of the 3AM War Room
A pragmatic, repeatable way to turn “it’s slow” into a measurable bottleneck, a safe fix, and a regression-proof rollout—across the usual suspects: DB, CPU/GC, N+1s, caching, and async backlogs.
If you can’t reproduce it, measure it, and roll it out safely, you didn’t optimize—you just changed code.Back to all posts
Related Resources
Key takeaways
- A performance playbook is a **repeatable incident-to-fix workflow**: baseline → isolate → change → validate → guardrail.
- If you don’t define **p95/p99, error rate, and saturation** up front, you’ll ship “optimizations” that don’t move customer pain.
- Most real-world slowdowns are boring: **bad SQL, missing indexes, N+1s, chatty services, GC pressure, cache misses, or overloaded queues**.
- Treat perf work like production work: **canaries, dashboards, rollback plan, and regression tests**.
- When the codebase is AI-assisted or legacy, run a **GitPlumbers Code Audit** or **Automated Insights** first to find structural perf traps fast.
Implementation checklist
- Define the user-facing symptom in one line (endpoint / job / page + p95/p99).
- Pick a target and a budget (e.g., p95 800ms → 350ms) and confirm it matches an SLO.
- Instrument traces + RED metrics (Rate, Errors, Duration) and saturation (CPU, I/O, pool utilization).
- Reproduce under load (k6/wrk) with production-like data shape.
- Identify the dominant cost center (DB time vs CPU vs network vs lock contention).
- Apply the smallest change that moves the needle; validate with before/after numbers.
- Ship with canary + dashboards + rollback; add a perf regression test to CI.
- Document the playbook and add “tripwires” (alerts on p95/p99 + saturation).
Questions we hear from teams
- What performance metrics should we standardize on across services?
- At minimum: **p95/p99 latency**, **error rate**, **throughput (RPS/QPS)**, and **saturation** (CPU, memory, DB pool usage, queue depth). Add one business-aligned metric per critical flow (e.g., checkout completion time).
- How do we avoid “optimizing” the wrong thing?
- Start with a baseline tied to an SLO and isolate the dominant cost center via traces (DB vs CPU vs downstream vs queue). If you can’t show a before/after improvement under comparable load and data shape, don’t ship it.
- When is scaling up acceptable vs real optimization?
- Scale when you’ve validated the system is efficient but under-provisioned (clear saturation, linear improvement with resources). Optimize when you see wasted work: N+1s, excessive serialization, lock contention, cache misses, or poor indexing.
- What’s the fastest way to triage a slow Postgres-backed endpoint?
- Enable `pg_stat_statements`, identify top queries by total time, run `EXPLAIN (ANALYZE, BUFFERS)` on the worst offender with realistic params, then fix indexes/query shape. Validate via traces and a repeatable load test.
Ready to modernize your codebase?
Let GitPlumbers help you transform AI-generated chaos into clean, scalable applications.