What’s the simplest way to start if we’ve never done proper load testing?

Pick one journey (checkout), define p95/error SLOs, write a k6 arrival-rate test, and build one Grafana dashboard that shows p95, error rate, and orders/minute. Run a 15-minute ramp + 10-minute soak. Fix one bottleneck. Repeat.

How do we avoid polluting production metrics with load test traffic?

Tag requests with headers (X-LoadTest), trace IDs, or a dedicated tenant ID. Filter in PromQL via labels and send load-test spans to a separate trace index. Always annotate dashboards and time windows.

Do we test in prod or staging?

Both. Staging for breakpoints and failure injection. Prod for low-traffic canary and synthetic checks to validate real dependencies (CDN, Auth, Payments). Never exceed agreed budgets in prod; use allowlists and off-peak windows.

What if our bottleneck is a third-party API?

Rate-limit upstream calls, batch when possible, cache responses with TTLs, and add circuit breakers/outlier detection. Negotiate higher SLAs and concurrency limits informed by your measurements. Model their fail modes in tests.

How do we quantify business impact from performance work?

Run A/B canaries with the same workload and compare conversion, revenue/order, and drop-off. Use statistical tests where possible. Tie infra changes to $ by multiplying conversion lift by traffic and AOV. Show cost reductions from right-sized infra.

Performance-optimization · Oct 6, 2025 · 10 minute read

Stop Chasing RPS: Load Tests That Protect p95, Revenue, and Sleep

Design load tests around user journeys and SLOs, not vanity throughput. Validate behavior under stress, prove business impact, and ship safer.

Eli Navarro

Principal Engineer, GitPlumbers

20 years building and rescuing distributed systems at scale. Ex-Netflix SRE, ex-Stripe performance team, observability nerd, and habitual p95 breaker.

“If your load test doesn’t move a business metric, it’s theater.”

Back to all posts

Stop Chasing RPS: Load Tests That Protect p95, Revenue, and Sleep

Key takeaways

Implementation checklist