The AI Copilot That Fell Over at 9:03 AM: How GitPlumbers Made It Boring Again
A real stabilization story: an AI-assisted customer-facing app, a Kubernetes cluster under stress, and the unglamorous fixes that cut error rates, cost, and on-call pain.
“Under real load, AI features don’t ‘scale later.’ They amplify every missing guardrail—retries, fan-out, and cost—until your pager becomes the product.”Back to all posts
Key takeaways
- If your AI feature has no explicit backpressure, it will melt your DB and your budget the first time customers show up.
- Most “LLM latency” is actually your own pipeline: retries, fan-out, retrieval, and missing caches.
- You can’t stabilize what you can’t see—distributed tracing with `OpenTelemetry` is non-negotiable for AI-assisted apps.
- RAG needs guardrails: timeouts, top-k caps, prompt/version pinning, and relevance thresholds prevent garbage-in/garbage-out cascades.
- Treat LLM calls like any other flaky dependency: circuit breakers, bulkheads, and graceful degradation beat heroic on-call firefighting.
Implementation checklist
- Define a user-facing SLO (latency + correctness proxy) and wire it to alerting.
- Instrument end-to-end traces: request → retrieval → LLM → DB writes.
- Add explicit rate limits per tenant and per endpoint.
- Cap fan-out (`topK`, chunk count) and enforce timeouts on retrieval and LLM calls.
- Introduce caching for embeddings/results where it’s safe and measurable.
- Protect your DB: pool sizing, query indexes, and async job isolation.
- Run a load test that matches real traffic shapes (bursts + long tails), not a toy RPS number.
- Ship fixes behind flags and canary them; don’t “big bang” AI changes on Friday.
Questions we hear from teams
- What made this app “AI-assisted” versus a normal web service?
- The critical path included retrieval (vector search + chunking), multiple external LLM calls, and post-processing that wrote back to `PostgreSQL`. That combination introduces long-tail latency, fan-out, and cost coupling that typical CRUD endpoints don’t have.
- Did you change LLM providers or models to get these results?
- No. We improved reliability primarily through backpressure, timeouts, circuit breakers, retrieval caps, and observability. Model/provider changes can help, but they don’t fix retry storms, DB pool exhaustion, or missing SLOs.
- How did you reduce hallucinations without a big ML project?
- We tightened the RAG pipeline: lower `topK`, relevance thresholds, context token caps, and prompt/version pinning. Most “hallucinations” we saw were retrieval and prompt-drift issues, not model mysticism.
- What’s the minimum tooling you need to do this kind of stabilization?
- `OpenTelemetry` (traces), `Prometheus` + `Grafana` (metrics), a load tool like `k6`, and the ability to ship small changes behind flags (GitOps with `ArgoCD` helps). Without tracing, you’ll burn days guessing.
Ready to modernize your codebase?
Let GitPlumbers help you transform AI-generated chaos into clean, scalable applications.