The Promo Engine That Blocked a Holiday Launch — And the 6‑Week Modernization That Freed It

Situation: A blocked launch at exactly the wrong time

If you’ve survived a retail holiday season, you know the smell of fear in a change review. This was a top‑10 e‑commerce player with a flagship promo campaign tied to TV buys and exclusive SKUs. The promotion engine lived deep in a Java/Spring 3.x monolith talking to Oracle 11g with a vendor ETL hanging off the side. p99 latency spiked to 2.8s under load tests, and any promo rule change required a redeploy during a change freeze. The CMO had a date circled in red; engineering had a pager budget dripping away.

I’ve seen this movie. Big promises, bigger rewrites, then an all‑hands rollback at 2 a.m. We did the opposite.

• Constraints

  • PCI scope, limited prod access, and a hard change freeze in 8 weeks
  • Spend cap: no new persistent cloud services beyond what FinOps had already approved
  • Non‑negotiables: launch date, existing checkout path, and zero downtime

• Signals

  • Synthetic tests showed promo API error spikes alongside GC events
  • Jenkins was a snowflake; last good restore unknown
  • “Helpful” AI‑generated patches sprinkled through the codebase—classic vibe coding: null guards without semantics, accidental O(n²) loops, copy‑pasted SQL

We were brought in to modernize just enough to make the date. No revolutions. No 12‑factor TED talk. Ship, safely.

What was actually broken (and why it matters)

The promo engine was married to the monolith’s session model. Every request did a read‑modify‑write against Oracle, recalculated eligibility, then fanned out to a third‑party pricing API. Under load, the GC paused, the pool starved, and retries stampede‑amplified the whole mess. Also: no feature flags, no circuit breakers, and logging was a print‑the‑world situation.

Why this matters:

• Business: A blown launch would nuke a quarter’s revenue and marketing trust. We costed the downside at mid seven figures.
• Engineering: Teams were paralyzed. Without guardrails, any fix risked a bad deploy during freeze.
• Operations: MTTR was 3.5 hours. No traces, no SLOs, and dashboards you had to squint at.

I’ve watched teams throw microservices at this and drown in retries. The better move: isolate the critical path, give it an escape lane, and add real control surfaces.

The six‑week plan we actually ran

We scoped a strangler‑fig carve‑out around promo calculation while leaving checkout intact. The minimum viable modernization looked like this:

1. Map the critical path, define SLOs: p95 <= 500ms, error rate < 0.5%, availability 99.95% during peak.
2. Stand up a parallel data path via Debezium CDC from Oracle to Postgres 14.
3. Implement a stateless promo service (Java 17, Spring Boot 3) with a read‑through Redis cache and Resilience4j circuit breakers.
4. Gate with OpenFeature flags. Rollout via Argo Rollouts canaries. Manage infra with Terraform. Deploy with ArgoCD under GitOps.
5. Add OpenTelemetry traces, Prometheus metrics, and actionable SRE‑grade alerts.
6. Rehearse failure: chaos tests, backpressure, forced rollback.

The client staffed 4 engineers; we embedded 3 from GitPlumbers. No heroics, just boring, repeatable moves.

Architecture changes, not a re‑platforming

We did not yank everything into Kubernetes. We containerized only what we needed and ran it on their existing EKS footprint with room to spare.

• Data: Debezium mirrored promo_rules and customer_segments into Postgres. Writes stayed on Oracle; reads came from Postgres via the new service.
• Service: Stateless promo calc behind NGINX Ingress; horizontal pod autoscaler sized by RPS and p95 latency.
• Control: OpenFeature flags gave us per‑segment rollout; Argo Rollouts handled weighted canary with automatic rollback.
• Resilience: Circuit breakers around the third‑party pricing API and the vendor ETL. Fast‑fail and degrade gracefully.

# argo-rollouts canary with metric-based analysis
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: promo-service
spec:
  replicas: 6
  strategy:
    canary:
      canaryService: promo-svc-canary
      stableService: promo-svc-stable
      steps:
        - setWeight: 5
        - pause: {duration: 5m}
        - setWeight: 25
        - pause: {duration: 10m}
        - analysis:
            templates:
              - templateName: error-rate-check
                args:
                  - name: maxErrorRate
                    value: "0.5"
  selector:
    matchLabels:
      app: promo
  template:
    metadata:
      labels:
        app: promo
    spec:
      containers:
        - name: promo
          image: registry/promo:1.3.7
          ports:
            - containerPort: 8080
          env:
            - name: OTEL_EXPORTER_OTLP_ENDPOINT
              value: http://otel-collector:4317

// Resilience4j circuit breaker + bulkhead around pricing API
var cb = CircuitBreaker.ofDefaults("pricing");
var bulkhead = Bulkhead.ofDefaults("pricing");
Supplier<Price> supplier = () -> pricingClient.getPrice(sku, customerId);
Supplier<Price> guarded = Decorators.ofSupplier(supplier)
    .withCircuitBreaker(cb)
    .withBulkhead(bulkhead)
    .withFallback(ex -> defaultPrice())
    .decorate();

Price price = guarded.get();

# Terraform (excerpt): EKS namespace + Redis with fixed budget
module "promo_ns" {
  source = "terraform-aws-modules/kubernetes-addons/aws"
  eks_cluster_id = var.cluster_id
  addons = ["metrics_server"]
}

resource "helm_release" "redis" {
  name       = "promo-redis"
  repository = "https://charts.bitnami.com/bitnami"
  chart      = "redis"
  namespace  = "promo"
  values = [file("helm/redis-values.yaml")]
}

// Debezium connector (excerpt)
{
  "name": "oracle-promo-connector",
  "config": {
    "connector.class": "io.debezium.connector.oracle.OracleConnector",
    "database.server.name": "oracleprod",
    "database.hostname": "oracle.internal",
    "database.user": "cdc_user",
    "database.password": "****",
    "database.dbname": "ORCL",
    "table.include.list": "PROMO.PROMO_RULES,PROMO.CUSTOMER_SEGMENTS",
    "snapshot.mode": "initial",
    "tombstones.on.delete": "false",
    "transforms": "unwrap",
    "transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState"
  }
}

Implementation notes from the trenches

A few things we did that made this boring in the best way:

• SLOs first, dashboards second. We pinned SLOs and built Prometheus + Grafana views to match. No vanity charts.

# PrometheusRecordingRules: error rate and latency windows
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: promo-slos
spec:
  groups:
    - name: promo-service
      rules:
        - record: job:promo_error_rate:5m
          expr: sum(rate(http_requests_total{job="promo",status!~"2.."}[5m]))
                / sum(rate(http_requests_total{job="promo"}[5m]))
        - record: job:promo_latency_p95:5m
          expr: histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{job="promo"}[5m])) by (le))

• Feature flags over config toggles. OpenFeature let us ramp by customer segment and SKU family. We shipped dark, then dialed weight.
• Cache correctness. Redis TTLs tied to rule versions; cache key included rule_version to avoid subtle stale reads.
• AI code triage. We yanked two “AI‑assisted” patches that quietly introduced quadratic scans in eligibility checks. Classic AI hallucination: code that compiles and vibes, but melts under load. We did a quick vibe code cleanup and added property‑based tests.

// Property-based test with fast-check to kill quadratic scans
test.prop([fc.array(fc.record({sku: fc.string(), seg: fc.string()}))])(
  'eligibility check scales roughly linear',
  (input) => {
    const t0 = performance.now();
    const result = eligibility(input);
    const t1 = performance.now();
    expect(t1 - t0).toBeLessThan(1.5 * input.length + 5);
  }
);

• Runbooks and abort conditions. We wrote a one‑pager: when error budget burns >2% in 15m or p95 > 700ms, Argo auto‑rolls back.

Results (the numbers that matter)

We didn’t win style points; we won the launch.

• Launch shipped on time with the new promo path carrying 82% of traffic by end of day 1; 100% by day 3
• Latency: p99 dropped from 2.8s to 350ms (‑87%); p95 at 220ms during peak
• Stability: error rate <= 0.3% during peak; zero Sev‑1 incidents; 0 rollbacks after canary stabilized
• MTTR: from 3.5h to 25m (7x improvement) thanks to traces and guardrails
• Throughput: sustained 4.2k RPS per AZ with headroom; HPA scaled to 12 replicas at peak, back to 4 off‑peak
• Costs: infra spend flat; Redis + extra EKS nodes offset by shorter GC pauses and reduced Oracle pressure
• Org: marketing got their knobs (flags), and engineering got their nights back

The exec post‑mortem was mercifully boring. That’s the goal.

What didn’t work and what we’d change next time

I wish I could say it was flawless. A few scars worth sharing:

• CDC lag: Debezium fell behind during a vendor ETL batch. We added a backpressure hint to slow canary growth when lag > 5s.
• Schema drift: A surprise VARCHAR2 length change broke a parser. We added CDC schema‑change alerts.
• Pricing API flaps: Circuit breakers saved us, but the fallback price annoyed finance for certain SKUs. Next time: pre‑compute fallback bands.
• Istio? We skipped a full mesh. Under this timeline, the value wasn’t there. If you already run Istio well, great; otherwise, keep it boring.

The playbook you can reuse tomorrow

If you’re staring at a date you can’t move and a monolith you can’t rewrite, steal this sequence:

1. Write SLOs that matter to revenue (p95 latency, error rate). Publish them.
2. Map the critical path. Add kill switches and feature flags before code changes.
3. Stand up CDC to a read‑optimized store. Avoid schema freezes.
4. Extract the smallest stateless service that pays for itself. Add caching and circuit breakers.
5. GitOps your deploys with canaries and auto‑rollback thresholds. Pre‑agree on abort conditions.
6. Instrument traces before tuning. You can’t optimize a ghost.
7. Rehearse failure in staging with chaos and realistic load.

If you’ve got AI‑generated code in the path, triage it early. We do AI code refactoring and code rescue all the time—it’s usually the cheap win that drops latency without new boxes.

You don’t need a platform. You need one lane that can pass traffic.

Where GitPlumbers fits

We’re not going to sell you a silver bullet or a quarterly platform roadmap. We do targeted modernization to unblock business outcomes: ship dates, SLOs, revenue moments. In this engagement, that meant a strangler carve‑out, GitOps, and enough observability to sleep at night.

If this sounds familiar, let’s talk. Bring us your deadline and your pager history; we’ll bring boring, proven moves.