How do I prevent users from switching A/B variants mid-conversation?

Use deterministic assignment based on an experiment key and a stable user/session ID. Propagate the assigned variant in headers or conversation state across services. Do not reassign on each request.

What metrics should be primary for a customer support bot?

Containment rate (no human handoff) should be primary. Track guardrail violation rate and cost per resolved conversation as constraints. Latency is a separate SLO with paging alerts.

How do I test new models without risking production?

Run offline evals to screen out bad candidates, then start a 1–5% canary using feature flags or Argo Rollouts. Add strict guardrails and a kill switch. Ramp only if primary metrics improve and constraints hold.

How do I detect model drift from providers like OpenAI/Anthropic?

Pin model versions when available. Maintain a smoke test suite of canonical prompts and expected patterns; run hourly and alert on deltas in latency, token usage, or output schema/length. Track response distribution changes (e.g., average tokens) over time.

What’s the fastest way to add observability to my LLM stack?

Wrap your LLM client with OpenTelemetry spans. Emit provider, model, version, token counts, and cost. Export to Prometheus (metrics) and Tempo/Honeycomb (traces). Add a p95 latency alert and a guardrail violation alert on day one.

Ai-delivery · Oct 10, 2025 · 9 minute read

A/B Testing LLMs in Production Without Burning Customers

The hard-won blueprint for shipping LLM experiments with real observability, safety guardrails, and metrics that actually move the business.

R. J. Morris

Partner, GitPlumbers

20-year engineering leader. Led platform/SRE at late-stage startups and Fortune 100s. Survived the monolith-to-microservices wave, Kubernetes adoption pain, and now production LLMs. I build systems that ship reliably and sleep at night.

Speed with a seatbelt: ship LLM experiments fast without betting the company on a hunch.

Back to all posts

A/B Testing LLMs in Production Without Burning Customers

Key takeaways

Implementation checklist