Our CI already takes 40 minutes. What’s the first lever to pull?

Selective execution. Use path-based filters or a build graph (Bazel/Nx/Gradle) to run only impacted tests. That alone usually cuts runtime 30–60%. Then shard long-running suites and cache dependencies.

Do we really need e2e tests if we have contract tests?

Yes, but keep them sparse. Contract tests protect interfaces; e2e validates cross-service workflows and deployment wiring. Limit to business-critical happy paths and run them against ephemeral envs to avoid flakes.

How do we measure CFR without gaming it?

Automate it. Tag every prod deployment and every incident with `deploy_sha`. Count incidents with customer impact or rollbacks within a window (24–72 hours). Report weekly, not per-PR, to avoid sandbagging.

Won’t canaries slow us down?

Not if you right-size steps and bake times. A 10% → 50% two-step canary with 1–2 minute pauses adds ~3–5 minutes but removes hours of MTTR. Net lead time improves because fewer rollbacks.

Release-engineering · Oct 27, 2025 · 9 minute read

Stop Shipping Regressions: The Test Gauntlet That Drops Change Failure Rate Without Killing Lead Time

What actually catches regressions early without turning your CI into molasses—and how to measure it with CFR, lead time, and recovery time as the north-star metrics.

Back to all posts

Stop Shipping Regressions: The Test Gauntlet That Drops Change Failure Rate Without Killing Lead Time

Key takeaways

Implementation checklist