Rollback First: The Boring-Friday Deploy Playbook
Design your delivery system so bad releases self-revert, metrics improve, and on-calls sleep through Friday nights.
If rollback isn’t one click, it’s a fantasy. Design it that way and Friday deploys get boring fast.Back to all posts
Key takeaways
- Design your system to prefer rollback over heroics: canary, flags, and immutable artifacts beat hotfix culture.
- Use change failure rate, lead time, and recovery time as non-negotiable scorecards tied to release policies.
- Rollback patterns aren’t one-size-fits-all: app, infra, and DB need different mechanisms that work together.
- Automate detection and abort: Argo Rollouts + Prometheus + Sentry can self-revert within minutes.
- Checklists are your scaling function: preflight, cutover, rollback, and postmortem steps must be explicit and rehearsed.
Implementation checklist
- Maintain immutable, versioned artifacts with provenance; keep at least N rollback candidates hot in the registry.
- Implement canary or blue/green with automated metrics-based abort (Prometheus/Kayenta/Argo Rollouts).
- Gate risky code paths behind kill-switch flags (OpenFeature/LaunchDarkly) and default them off.
- Use expand/contract DB migrations (Flyway/Liquibase) and online schema change tools (gh-ost/pt-osc).
- Pre-merge CI verifies rollbackability: schema backward-compat, migration plan, and artifact promotions.
- Release trains include rehearsed rollback playbooks and a single-button undo (Helm/Argo/Kubectl).
- Observability budgets wired to deployments: error rate, latency, and business KPIs drive automated rollback.
- Run weekly game days that drill rollback of the latest release with production-like data.
Questions we hear from teams
- Rollback vs roll forward: which should we prefer?
- Prefer rollback for acute production incidents—it's faster and safer because you’re moving to a known-good. Roll forward is great for known bug fixes or when the issue is data-dependent and a new build is safer. Mature teams have both, but your first response to a red canary should be rollback.
- How do we handle database rollbacks?
- You don’t. You design changes to be backward compatible (expand/contract), and you pause backfills if needed. Avoid destructive changes until the new code has been live for at least one full release cycle. If you must revert data, use journaling and idempotent backfill jobs so you can replay.
- Do feature flags add too much complexity?
- They add complexity, but controlled complexity. Keep flags short-lived, use naming conventions, auto-expire them, and include kill-switch checks in code review. The reduction in MTTR more than pays for the overhead.
- We’re not on Kubernetes. Does this still apply?
- Yes. On ECS use CodeDeploy blue/green. On Lambda use versions/aliases. On VMs use Nginx/HAProxy weighted upstreams. The principles—immutable artifacts, traffic-shifted rollout, metrics-based abort—are platform agnostic.
- How do we prove improvement to leadership?
- Show CFR, lead time, and MTTR trends alongside incident counts and customer KPIs. Tie them to dollars: fewer incidents reduce support load and churn; faster recovery protects revenue during spikes. Put these in your monthly business review.
Ready to modernize your codebase?
Let GitPlumbers help you transform AI-generated chaos into clean, scalable applications.