The Six‑Week Save: How “Just‑Enough” Modernization Unblocked a Regulated Launch Without Torching Prod

The launch that was dead on arrival

Mid‑market fintech. Regulated payouts feature. Board‑committed date six weeks out. On paper, simple: extend existing payout rails with new compliance checks and a regional data residency requirement. In reality, their 2014-era .NET Framework 4.6 monolith, Jenkins freestyle job zoo, and a growing pile of AI‑generated “helper” classes made releases a coin flip.

We got the 7 a.m. call: “If we push now, we’ll miss quarter. If we don’t, we’ll breach the contract.” Been there. The CTO didn’t want platitudes; they needed a safe path to ship without rewriting the world or waking auditors.

Constraints we had to respect:

• Regulatory: SOC 2 Type II in progress, PCI scoped; change control had to be auditable.
• Infra: EKS 1.25 already running; prod in us-east-1, new data has to land in eu-west-1.
• Budget/time: Six weeks. No extra headcount. No greenfield dreams.
• Risk: Two prior rollback Fridays burned the team. One more outage and legal gets involved.

I’ve seen teams try to “microservice their way out” here and faceplant. We cut scope to modernization moves that reduced release risk and latency to ship—nothing else.

What we found in week one

We ran a tight, three‑day assessment. Not a 70‑page PDF—just the minimum to de‑risk the launch.

Highlights (or lowlights):

• Release unpredictability: 47 Jenkins jobs with bespoke Bash; half broke on any agent change. Lead time averaged 14 days. Change failure rate 22%.
• AI‑generated code drift: ChatGPT‑spawned data mappers and “DTO fixers” duplicated logic and did unchecked JSON parsing. One helper logged PII to stdout during errors. Vibe coding at its finest.
• Observability gap: No tracing. Logs lived on disk until logrotate ate them. Alerts were “CEO Slack ping.”
• Secrets: .env files in S3, hand‑copied to nodes. Rotations caused silent auth failures.
• Schema drift: Terraform managed clusters, but teams clicked in AWS to “fix” things; terraform plan was a horror show.

The immediate blocker wasn’t architecture. It was the inability to deploy safely and know if we were burning the error budget.

The modernization we actually did (and what we cut)

We made a rule: no new microservices unless it removes a hard blocker. We kept the monolith but carved a thin seam.

What we did in 21 days:

1. Feature flags, day 2: Wrapped new payout flows with LaunchDarkly flags (payouts.eu.compliance). Default OFF. This turned risky deploys into safe config changes.
2. GitOps for prod only: Introduced ArgoCD to reconcile production manifests from a deploy-prod repo. Staging stayed on Jenkins for a week to avoid shock.
3. Canary + rollback: Used Argo Rollouts to canary the monolith Deployments at 5% → 25% → 50% → 100% with automated rollback on SLO burn or elevated 5xx.
4. Telemetry that matters: Added OpenTelemetry to the monolith’s critical endpoints and piped traces/metrics to Prometheus/Grafana. Published two SLOs: Availability 99.9% and p95 latency < 300ms for payout authorize.
5. Secrets fixed at the source: Moved to ExternalSecrets backed by AWS Secrets Manager. Killed the .env ritual.
6. Just‑enough refactor: Strangled two AI‑generated mappers into a single, tested component. No heroic rewrite; this removed the top two crashers.
7. Compliance breadcrumbs: Every deploy ran through GitHub Actions, gated by change approvals, and ArgoCD created an immutable audit trail. Auditors love receipts.

What we cut:

• No wholesale .NET 8 rewrite. We did lift one perf‑critical path into a .NET 8 sidecar via gRPC, behind a flag, to de‑risk latency. Everything else stayed.
• No Istio mesh rollout. We used NGINX Ingress + Argo Rollouts for traffic splitting.
• No Terraform full‑court press. We pinned cluster config and created a “no‑clicks in prod” rule. Terraform refactor can wait till after revenue lands.

The pipeline and GitOps changes that mattered

We killed Jenkins job roulette for production and moved to a single, composable pipeline in GitHub Actions, with ArgoCD pulling from a deploy repo.

Here’s the trimmed‑down CI that built, scanned, and cut a signed release:

ame: monolith-ci
on:
  push:
    branches: [ main ]
  workflow_dispatch:
jobs:
  build-test:
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-dotnet@v4
        with:
          dotnet-version: '6.0.x'
      - name: Restore & Test
        run: |
          dotnet restore
          dotnet test --collect:"XPlat Code Coverage" --logger trx
      - name: Build Docker
        run: |
          docker build -t ghcr.io/acme/monolith:${{ github.sha }} .
      - name: Scan Image
        uses: aquasecurity/trivy-action@0.24.0
        with:
          image-ref: ghcr.io/acme/monolith:${{ github.sha }}
          severity: HIGH,CRITICAL
      - name: Push Image
        run: |
          echo $CR_PAT | docker login ghcr.io -u $GITHUB_ACTOR --password-stdin
          docker push ghcr.io/acme/monolith:${{ github.sha }}
      - name: Create release tag
        run: |
          git tag -a rel-${{ github.run_number }} -m "release"
          git push origin rel-${{ github.run_number }}

Deployment moved out of CI and into GitOps. ArgoCD watched a dedicated repo that templated the image tag and rollout strategy.

# k8s/monolith-rollout.yaml
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: monolith
spec:
  strategy:
    canary:
      steps:
        - setWeight: 5
        - pause: {duration: 120}
        - setWeight: 25
        - pause: {duration: 180}
        - setWeight: 50
        - pause: {duration: 300}
      trafficRouting:
        nginx: {}
      analysis:
        templates:
          - templateName: error-rate
        args:
          - name: service-name
            value: monolith
  selector:
    matchLabels: { app: monolith }
  template:
    metadata:
      labels: { app: monolith }
    spec:
      containers:
        - name: monolith
          image: ghcr.io/acme/monolith:rel-123
          envFrom:
            - secretRef: { name: monolith-secrets }

And the AnalysisTemplate that gated the canary with Prometheus:

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: error-rate
spec:
  args:
    - name: service-name
  metrics:
    - name: http-5xx-rate
      interval: 60s
      successCondition: result < 0.02
      failureLimit: 2
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            sum(rate(nginx_ingress_controller_requests{exported_service=~"{{args.service-name}}",status=~"5.."}[5m]))
            /
            sum(rate(nginx_ingress_controller_requests{exported_service=~"{{args.service-name}}"}[5m]))

Secrets stopped being copy‑paste adventures with ExternalSecrets:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: monolith-secrets
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets
    kind: ClusterSecretStore
  target:
    name: monolith-secrets
  data:
    - secretKey: DB_CONNECTION
      remoteRef:
        key: /prod/monolith/db-connection

Compliance win: ArgoCD gave an append‑only deploy history with who/what/when. Auditors stopped asking for screenshots of Jenkins logs.

Risk‑managed rollout: flags, canaries, and SLOs

We refused to ship without two SLOs and budget burn alerts. You don’t need a PhD—just measure the golden paths and wire rollback to them.

Minimal OpenTelemetry in the monolith (C#):

// Program.cs (.NET 6 hosting for the monolith)
using OpenTelemetry;
using OpenTelemetry.Metrics;
using OpenTelemetry.Trace;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddOpenTelemetry()
    .WithTracing(b => b
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddSource("Payments")
        .SetSampler(new TraceIdRatioBasedSampler(0.1))
        .AddOtlpExporter())
    .WithMetrics(b => b
        .AddAspNetCoreInstrumentation()
        .AddRuntimeInstrumentation()
        .AddPrometheusExporter());

var app = builder.Build();
app.MapPrometheusScrapingEndpoint();
app.Run();

Simple SLO burn alert (Prometheus rule):

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: payouts-slo
spec:
  groups:
    - name: slo.availability
      rules:
        - alert: PayoutsSLOBurn
          expr: (1 - sum(rate(http_requests_total{handler="/payouts/authorize",status=~"2..|3.."}[5m])) / sum(rate(http_requests_total{handler="/payouts/authorize"}[5m]))) > 0.001
          for: 10m
          labels:
            severity: page
          annotations:
            summary: "Payouts SLO burn rate high"

Feature flag wrapper around the risky flow (launch darkly pseudo‑C#):

if (ldClient.BoolVariation("payouts.eu.compliance", user, false))
{
    return await NewComplianceFlow(request);
}
return await LegacyFlow(request);

Rollout plan the team could run half‑asleep:

• Cut release. ArgoCD picks it up, canary begins at 5%.
• Watch Grafana SLO + error rate; Argo Rollouts auto‑pauses on threshold breach.
• If burn alerts or 5xx spike, hit argo rollouts abort monolith. Rollback happens in seconds.
• Once stable at 50%, flip the feature flag ON for a single EU tenant.
• If tenant telemetry is good for 24h, scale to 100% and expand the flag audience.

No heroics, no cowboy deploys. Just gates, signals, and reversibility.

Results: the numbers and the business impact

By week six, the feature launched to the first 10 EU customers and expanded to 100% of EU traffic by week eight. More importantly, the org could finally ship without crossing fingers.

Measurable outcomes:

• Lead time for changes: 14 days → 2 hours (median) to production.
• MTTR: 4 hours → 18 minutes, courtesy of canaries + fast rollback.
• Change failure rate: 22% → 4% over the next 30 days.
• p95 latency (authorize): 520ms → 230ms after the .NET 8 sidecar for the hot path.
• On‑call pages: 9/week → 2/week.
• Compliance: SOC 2 auditors accepted ArgoCD history as change evidence. Zero exceptions.

Business side:

• Launch date held. Contractual milestone met; no penalties.
• Pipeline: $3.1M in booked ARR tied to EU payouts within the quarter.
• Team morale: The Friday “deploy freeze” died. Engineers volunteered to own the next refactors.

The CTO’s note after launch said it best:

“We didn’t get a fairy‑tale rewrite. We got a system we can ship. That’s what we needed.”

What we’d do differently next time—and what you can steal on Monday

What I’d tweak:

• Start feature flags day zero, not day two. It paid back instantly.
• Bake SLOs into planning, not the last mile. Product leaders understood “error budget” faster than I expected.
• Move the second hot path to .NET 8 sooner; the gRPC seam pattern worked well.

What you can apply without calling us:

1. Identify the constraint. If deploy risk is the bottleneck, modernize the release path first.
2. Insert flags before refactors. Ship behind OFF, then stabilize.
3. Adopt GitOps where audit matters most: production. Backfill staging later.
4. Add the simplest SLOs for your golden paths and wire automated rollback.
5. Kill secrets drift with ExternalSecrets or your cloud’s KMS.
6. Delete AI‑generated “helpers” that log PII or duplicate logic. Do a vibe code cleanup pass.

If you’re staring down a board date with a 2014 monolith and some LLM‑written gremlins, you don’t need a moonshot. You need guardrails and reversibility. That’s the work GitPlumbers does every week.