What tools should I start with if I have nothing today?

Start simple: promptfoo in CI for golden sets; OpenTelemetry for traces; Prometheus + Grafana for metrics; LaunchDarkly for gating; Argo Rollouts for canary. Add Ragas/TruLens for grounding scores when you introduce RAG. Layer on OPA and Pydantic for guardrails.

How do I pick thresholds for pass/fail?

Back-solve from business KPIs. If a 2% hallucination rate leads to 10% ticket escalations, your threshold is below 2%. Shadow for a week to calibrate baselines, then set conservative gates for canary and tighten after you see stability.

Won’t all this slow us down?

It speeds you up after the second incident you avoid. We’ve seen teams ship weekly instead of quarterly because the harness catches regressions early and rollbacks are automatic. Your on-call sleeps better, too.

Ai-delivery · Oct 15, 2025 · 9 minute read

The Eval Harness That Keeps Your Gen Features Honest—Before, During, and After Release

Stop praying your AI features behave in prod. Instrument them like any other critical system and make them prove it every day.

Alex Kim

Partner, GitPlumbers

20 years building and rescuing distributed systems at scale. Led SRE and platform teams through the microservices wave, lived the pager life, and now helps companies ship AI features that don’t blow up in prod.

If you can’t parse it, you can’t trust it. If you can’t trace it, you can’t debug it.

Back to all posts

The Eval Harness That Keeps Your Gen Features Honest—Before, During, and After Release

Key takeaways

Implementation checklist