The Vibe‑Coded App That Kept Falling Over: A Production Rescue in 12 Days

The week it started failing in public

This was a seed-stage startup with a familiar story: fast product iteration powered by AI-assisted coding (what everyone calls “vibe coding”), a handful of paying customers, and a founder doing the classic split-brain routine—sales calls by day, hotfixes by night.

They shipped an app built on Next.js 14 + Node.js, Prisma, PostgreSQL, and a thin API layer deployed on AWS ECS Fargate. It worked—until it didn’t.

By the time GitPlumbers got pulled in, production looked like this:

• 3–6 customer-visible incidents per week (mostly 500 spikes)
• p95 API latency swinging from 450ms to 8–12s during peak usage
• Deploys that “succeeded” but silently broke background jobs
• A scary mix of copied snippets, half-wired retries, and no consistent error handling

The founder’s exact line: “We can’t raise the next round if the app randomly dies during demos.”

Also: investors were starting to ask for basic diligence artifacts—SOC2 trajectory, incident history, dependency risk—stuff vibe-coded repos rarely have ready.

What we found (a codebase that shipped, but wasn’t built to run)

First, definitions in plain English:

• Technical debt = the cost you pay later for shortcuts now (usually as incidents, slow shipping, or surprise rebuilds).
• SLO (Service Level Objective) = a concrete reliability target like “99.9% of requests succeed.”
• Code audit = a structured review of architecture, operational risks, security posture, and delivery health—not just “style feedback.”

We started with two things in parallel:

1. GitPlumbers Automated Insights (GitHub-integrated analysis) to quickly surface structural risks: dependency issues, security gaps, hot spots, and complexity.
2. A focused code audit to connect the findings to production behavior (logs, metrics, deploy history, and the actual incident timeline).

The top failure modes weren’t exotic. They were the kind of stuff you only learn after you’ve been paged at 2 a.m.:

• Unbounded retries in API handlers (classic: retrying DB calls inside a request without a cap)
• N+1 queries and missing indexes (Prisma makes it easy to ship slow queries fast)
• No idempotency on write endpoints (double-submits = duplicated rows = inconsistent state)
• Migrations run manually (deploys “worked” until a background worker hit a missing column)
• Secrets hygiene issues (a real .env had been committed early on; it was removed later, but… git never forgets)
• No end-to-end smoke test between “deploy” and “customers use it”

Automated Insights flagged the usual suspects:

• Outdated packages with known CVEs
• A few “god files” with high churn and high complexity
• Missing boundary validation (accepting arbitrary JSON into DB writes)

But the audit connected the dots: those issues mapped directly to incidents and churn risk.

Constraints (why we didn’t rewrite)

Here’s what made this a rescue, not a rebuild:

• Runway constraint: they had ~5 months of runway. A rewrite was a fundraising-death-spiral bet.
• Customer constraint: paying users were actively using the system. No “pause and rebuild.”
• Team constraint: 1 founder, 2 mid-level engineers. No staff to build a platform team.
• Risk constraint: investors wanted evidence of control: reliability plan, security posture, predictable delivery.

I’ve seen this fail when teams try to “clean everything.” You don’t need perfection. You need predictable operations.

So we agreed on a stabilization SLO and a short, brutal scope:

• Target SLO: 99.9% successful API requests for core user flows
• Reduce incidents from weekly to “rare”
• Make deploys boring

The intervention: 12 days, three workstreams, zero heroics

We ran this like an incident-response-driven refactor. Three workstreams, executed in parallel.

1) Observability that answers “why” in under 5 minutes

They had logs, but not usable logs. No request IDs. No consistent error taxonomy. No way to correlate spikes to releases.

We implemented:

• Structured logging with pino
• Request correlation via x-request-id
• Error normalization (so 500 means something)
• Tracing hooks (lightweight OpenTelemetry setup)

// middleware/requestContext.ts
import pino from "pino";
import { randomUUID } from "crypto";

export const logger = pino({ level: process.env.LOG_LEVEL ?? "info" });

export function withRequestContext(req, res, next) {
  const requestId = req.headers["x-request-id"] ?? randomUUID();
  res.setHeader("x-request-id", requestId);

  req.log = logger.child({ requestId, path: req.path, method: req.method });
  next();
}

This wasn’t about “pretty dashboards.” It was about MTTR (Mean Time To Recovery). When something broke, we wanted the on-call to answer:

• What changed?
• Which endpoint is failing?
• Is it DB, dependency, or code path?

2) Production safety rails: timeouts, idempotency, validation

The vibe-coded pattern we see constantly: optimistic code paths with no guardrails.

We added:

• Hard timeouts on outbound calls
• Consistent input validation with zod
• Idempotency keys for write endpoints (especially webhook-like flows)

// api/createCheckout.ts
import { z } from "zod";
import { prisma } from "../db";

const schema = z.object({
  userId: z.string().uuid(),
  planId: z.string().min(1),
  idemKey: z.string().min(16)
});

export async function createCheckout(req, res) {
  const { userId, planId, idemKey } = schema.parse(req.body);

  const existing = await prisma.checkout.findUnique({ where: { idemKey } });
  if (existing) return res.status(200).json(existing);

  const created = await prisma.checkout.create({
    data: { userId, planId, idemKey }
  });

  return res.status(201).json(created);
}

This one change alone eliminated an entire class of “duplicate row” bugs that were previously showing up as random downstream failures.

3) Delivery hardening: CI/CD that fails fast (before customers do)

Their deploy pipeline was basically: “merge to main and hope.” We’ve all been there. It’s fine until it’s not.

We updated GitHub Actions so merges run:

• lint + typecheck
• unit tests (light, fast)
• migration check
• a smoke test that hits the deployed API

# .github/workflows/ci.yml
name: ci
on: [pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: npm
      - run: npm ci
      - run: npm run lint
      - run: npm run typecheck
      - run: npm test
      - run: npx prisma migrate diff --from-schema-datamodel prisma/schema.prisma --to-url "$DATABASE_URL" --exit-code

The migration diff check was key: it prevented the “deploy succeeded but jobs crash later” failure.

We also added a basic load test for one critical journey using k6:

k6 run --vus 20 --duration 60s scripts/smoke_checkout.js

Not because load testing is trendy—because it catches the exact kind of “works on my laptop” latency explosions that kill retention.

The targeted debt payoff (the 5 fixes that moved the needle)

We didn’t “refactor everything.” We paid down debt where it directly reduced incidents and cost.

1. Database indexes for real queries

   • We pulled slow query logs and added targeted indexes.
   • Result: p95 on the worst endpoint dropped from ~9s to ~1.1s.

2. Killed unbounded retries and added circuit breakers

   • Retries became capped with jitter.
   • Timeouts were enforced so requests fail predictably instead of hanging.

3. Made background jobs idempotent and observable

   • Jobs were reworked to be safely retryable.
   • Added dead-letter handling (even a simple “failed_jobs” table is better than vibes).

4. Secret scanning + rotation + repo hygiene

   • Ran secret detection, rotated exposed keys, and enabled GitHub secret scanning.
   • Removed shared “admin” credentials and tightened IAM policies.

5. Dependency risk cleanup

   • Upgraded a handful of high-risk packages flagged by Automated Insights.
   • Removed abandoned libraries that were only “working” because nobody touched them.

Results (measurable outcomes in two weeks)

This is the part founders care about: did it buy runway and confidence?

Within 12 days:

• Incidents: dropped from 3–6/week to 0–1/week (and the remaining ones were non-customer-facing)
• MTTR: improved from ~2–4 hours to ~25 minutes (because logs/traces finally told the truth)
• p95 latency (core API): improved from multi-second spikes to <1.5s under normal load
• Error rate: 5xx rate during peak went from ~2.8% to ~0.3%
• Cloud cost: reduced by ~18% (less thrash, fewer runaway retries, right-sized task CPU)
• Shipping cadence: moved from “fear merges” to 2–3 safe deploys/week

The investor-diligence win was real too:

• A clear reliability narrative: SLO, changes made, and before/after metrics
• A documented remediation plan for remaining debt (with effort and risk)
• Security posture improvements that didn’t require a compliance theater rewrite

What to do if your vibe-coded app is wobbling

I’ve seen this movie: the app is “done,” customers are paying, and the team is terrified to touch anything because every change breaks something unrelated.

If that’s you, here’s what actually works:

1. Stop debating rewrite vs refactor in the abstract. Define the SLO and measure the gaps.
2. Run Automated Insights to get fast signal on structural risk (dependencies, hotspots, security).
3. Book a code audit to translate that signal into an execution plan tied to incidents and business risk.
4. Fix the top failure modes first: timeouts, retries, idempotency, migrations, observability.
5. Assemble a fractional remediation team if you can’t afford senior hires yet. (This is exactly where GitPlumbers’ Team Assembly fits: you get the right seniors for the specific mess you have, not generic “more devs.”)

The goal isn’t “beautiful code.” The goal is an app you can trust in front of customers and investors.

If you want the same rescue path: run GitPlumbers Automated Insights on your repo, then book a code audit. You’ll get a prioritized risk map, a stabilization plan, and (if needed) a fractional team to execute without derailing product delivery.