The DR Plan That Survived a Breach: Policy to Guardrails, Checks, and Proofs

The 2 a.m. breach that broke the “DR-only” plan

We had a client with picture-perfect DR for region failover. Runbooks laminated, Route 53 failover tested, RDS snapshots like clockwork. Then 2 a.m. on a Tuesday: an engineer’s personal access token was pushed to a public repo during a rushed hotfix, and an attacker started siphoning data through a seemingly legit service account. Zero floods. No AZ outage. Just a quiet exfiltration through the front door.

The DR playbook didn’t mention token revocation, KMS key rotation, or terminating Okta sessions. The incident response doc existed, but it wasn’t wired into deployments, access, or observability. MTTR wasn’t measured for “revoked keys,” only “cluster back online.” That night we learned: if your DR plan doesn’t include security breach scenarios, you don’t have DR—you have wishful thinking.

Security incidents are disasters. Treat them with the same rigor, SLOs, and automation as failovers.

Make DR objectives security-aware

Classic DR talks RTO and RPO. That’s necessary, but during a breach you also need to prioritize containment and integrity over raw uptime.

• Add breach SLOs alongside RTO/RPO:
  • MTTD (mean time to detect) and MTTR for containment.
  • Blast radius SLO: max number of credentials or tenants exposed before containment.
  • Key rotation SLO: e.g., rotate compromised KMS key grants within 15 minutes.
  • Session revocation SLO: invalidate SSO sessions and API tokens within 10 minutes.
• Define minimal viable service: what you can run safely while secrets rotate and audit runs.
• Prioritize integrity over availability: better to go read-only or shed load than serve tampered data.
• Pre-approve compensating controls: feature flags to disable risky flows, circuit breakers to block outbound data egress, coarse-grained kill switches.

When leadership asks “are we up?”, you want to answer “we’ve contained the blast, revoked tokens, rotated keys, and restored read-only service to 90% of tenants” with metrics, not vibes.

Turn policy into guardrails you can’t ignore

Policies in PDFs don’t stop breaches; executable guardrails do. Here’s what actually works.

• Infrastructure policy as code with OPA/Conftest

# policy/terraform_s3.rego
package rules.s3

# Deny buckets without encryption or with public ACLs
violation[msg] {
  resource := input.resource_changes[_]
  resource.type == "aws_s3_bucket"
  exists(resource.change.after.acl)
  lower(resource.change.after.acl) == "public-read"
  msg := sprintf("S3 bucket %s has public ACL", [resource.address])
}

violation[msg] {
  resource := input.resource_changes[_]
  resource.type == "aws_s3_bucket_server_side_encryption_configuration"
  not resource.change.after.rule.apply_server_side_encryption_by_default.sse_algorithm
  msg := sprintf("SSE not enforced for %s", [resource.address])
}

Run it in CI with conftest test plan.json after terraform plan -out=plan && terraform show -json plan > plan.json.

• Kubernetes admission controls with Kyverno

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-signed-images
spec:
  validationFailureAction: Enforce
  rules:
    - name: verify-cosign
      match:
        resources:
          kinds: [Pod]
      verifyImages:
        - image: "ghcr.io/yourorg/*"
          key: "cosign.pub"
          attestations:
            - name: slsa-provenance
              predicateType: https://slsa.dev/provenance/v1

Block unsigned images and require SLSA provenance. No signature, no deploy.

• Org-wide safety rails with AWS SCPs

Deny dangerous actions at the root so a compromised role can’t nuke logs or turn off KMS:

{
  "Version": "2012-10-17",
  "Statement": [
    {"Effect": "Deny", "Action": ["kms:DisableKey", "kms:ScheduleKeyDeletion"], "Resource": "*"},
    {"Effect": "Deny", "Action": ["logs:DeleteLogGroup", "logs:DeleteLogStream"], "Resource": "*"}
  ]
}

• CI that fails closed

# .github/workflows/policy.yml
name: policy
on: [pull_request]
jobs:
  verify:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: terraform init && terraform plan -out=plan
      - run: terraform show -json plan > plan.json
      - uses: instrumenta/conftest-action@v1
        with: {files: plan.json}
      - name: Scan images
        run: |
          trivy fs --exit-code 1 --severity CRITICAL,HIGH .
      - name: Verify signatures
        run: cosign verify --key cosign.pub ghcr.io/yourorg/service:${{ github.sha }}

Your policy now stops bad infra, blocks unsigned code, and breaks builds that put you at risk.

Automated proofs, not screenshots

Auditors don’t want slide decks; they want evidence. You don’t want humans screenshotting dashboards at quarter-end, either. Produce signed, machine-verifiable proofs during CI/CD.

• Emit JSON evidence for each control and sign it with cosign.

# generate_evidence.sh
set -euo pipefail
mkdir -p evidence
jq -n --arg build "$GITHUB_SHA" --arg time "$(date -Iseconds)" \
  '{control:"S3 encryption enforced", build:$build, time:$time, status:"pass"}' > evidence/s3_encryption.json
cosign attest --predicate evidence/s3_encryption.json \
  --predicate-type https://example.com/controls/v1 \
  --key cosign.key ghcr.io/yourorg/service:${GITHUB_SHA}

• Store immutable proofs

  • Push attestations to an OCI registry or transparency log (Rekor).
  • Mirror to an append-only S3 bucket with object lock and AWS Backup.

• Aggregate control status

  • Use Security Hub, AWS Config, or OpenSearch dashboards to show “last-pass time” per control.
  • Set alerts if a control goes stale (no fresh evidence within SLO).

End result: when the auditor asks “prove encryption at rest,” you point to signed, time-stamped artifacts linked to commits and deployments.

Breach playbooks you can run, not read

In a breach, humans are slow and sleepy. Convert runbooks into idempotent scripts and make targets that can be run by on-call with minimal context.

• Rotate IAM access keys and update K8s secrets

# scripts/rotate_access_key.sh
set -euo pipefail
USER=$1
NEW=$(aws iam create-access-key --user-name "$USER" | jq -r .AccessKey.AccessKeyId)
SECRET=$(aws iam list-access-keys --user-name "$USER" | jq -r .AccessKeyMetadata[0].AccessKeyId)
aws iam delete-access-key --user-name "$USER" --access-key-id "$SECRET"

# update k8s secret
kubectl create secret generic api-creds --from-literal=AWS_ACCESS_KEY_ID=$NEW \
  --from-literal=AWS_SECRET_ACCESS_KEY=$(aws iam list-access-keys --user-name "$USER" | jq -r .AccessKeyMetadata[0].Status) \
  -o yaml --dry-run=client | kubectl apply -f -

kubectl rollout restart deploy -n apps backend

• Revoke SSO sessions and OAuth tokens

# Okta example (requires Okta API token)
USER_ID=$1
curl -s -X POST "https://yourorg.okta.com/api/v1/users/$USER_ID/lifecycle/revoke_sessions" \
  -H "Authorization: SSWS $OKTA_TOKEN" -H "Content-Type: application/json"

• Segment traffic fast
  • Pre-wire Istio/NGINX rules to block egress to sensitive endpoints on a feature flag.
  • Keep per-tenant allowlists and a kill switch to force read-only for impacted tenants.

Runbooks should log every action to a dedicated audit sink with request IDs. If you can’t run it safely in staging, it won’t save you at 2 a.m.

Drill it like you mean it

We run quarterly security GameDays with clients. The rules: production-like data, real tooling, and a stopwatch.

1. Announce a scenario: leaked CI token detected; attacker using a service account.
2. Start the clock. PagerDuty fires. On-call isolates affected apps via feature flags.
3. Revoke tokens and sessions. Rotate keys with scripts. Force redeploy with new secrets.
4. Query logs and DLP to confirm exfil window. Notify stakeholders.
5. Restore minimal service. Collect evidence artifacts.
6. Retrospect with metrics: time to isolate, revoke, rotate, restore.

Targets we like to see after two cycles:

• MTTD under 5 minutes via anomaly alerts (impossible travel, unusual egress, CI token use from new ASN).
• Session revocation within 10 minutes; credential rotation within 15 minutes.
• Read-only restore for 90% of tenants within 30 minutes.
• 95% controls covered by automated, signed evidence artifacts.

If your numbers aren’t improving, your guardrails aren’t where the work happens (hint: put them in CI and admissions, not Confluence).

Move fast with regulated data—safely

You don’t have to pick between HIPAA/GDPR and shipping velocity. You need a safety envelope.

• Data classification by default: tag resources and schemas with data_class=public/internal/confidential/regulated. Deny egress of regulated data to non-compliant paths.
• Ephemeral environments: per-PR namespaces with masked secrets; destroy on merge. Seed with synthetic or tokenized data.
• JIT and break-glass access: use Teleport/AWS IAM Identity Center for time-bound roles. Log every elevation; require manager + security approval for regulated datasets.
• DLP and schema guards in CI: block code that writes PII to logs; fail builds on accidental S3 public policies.

Example: enforce PII log blocking with semgrep and prevent prod deploy without approvals.

# .github/workflows/deploy.yml
name: deploy
on:
  push:
    branches: [main]
jobs:
  scan-and-deploy:
    permissions: {id-token: write, contents: read}
    environment: production
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: returntocorp/semgrep-action@v1
      - name: Require approvals for prod
        if: github.ref == 'refs/heads/main'
        run: |
          reviewers=$(gh api repos/${{ github.repository }}/environments/production/protection_rules | jq length)
          test $reviewers -ge 2
      - name: Deploy
        run: ./scripts/deploy.sh

And block PII-in-logs in K8s with Kyverno:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: forbid-pii-logs
spec:
  validationFailureAction: Enforce
  rules:
    - name: block-pii
      match:
        resources: {kinds: [Pod]}
      validate:
        message: "Containers must not set LOG_PII=true"
        pattern:
          spec:
            containers:
              - env:
                  - name: LOG_PII
                    value: "false"

The result: developers move fast within a guardrail that keeps regulators and customers happy.

What good looks like (and how GitPlumbers helps)

When we retrofit breach-ready DR at clients, we aim for:

• 90% of controls enforced pre-merge or at admission; changes that violate policy don’t land.
• Automated, signed evidence for all high-risk controls (encryption, provenance, access)
• Runnable runbooks with a measured rotation time under 15 minutes.
• Quarterly drills with improving SLOs and clear residual-risk reports.

I’ve seen the other version—binder full of policies, human approvals, screenshots. It always fails under real pressure. If you want help building the version that works, that’s what we do at GitPlumbers: turn policies into guardrails, checks, and proofs you can run before the pager goes off.

• Need to make this real? Bring us your current DR/IR docs and a week of pipeline access. We’ll map controls, encode guardrails, and run your first GameDay.
• Want to see examples? We’ve got case studies where we cut rotation MTTR by 70% and replaced auditor screenshots with signed attestations.

Ship safely, sleep better. The pager will still ring—but you’ll be ready.