Do I really need both dbt and Great Expectations?

Use dbt tests for schema-level and relational checks close to your SQL transforms; add Great Expectations for richer distribution and field-level validations. They’re complementary. Start with dbt tests; layer GE where drift risk is high.

Feast vs custom feature service?

You can roll your own, but you’ll reimplement registry, materialization, and online/offline parity. Feast gives you consistent definitions and pluggable stores (Redis/Dynamo/Bigtable) with minimal ceremony. Start with Feast; customize only where latency or scale demands.

Warehouse or lake for ML features?

Use a lake with Delta/Iceberg for time travel and cost control. If your org is warehouse-heavy (Snowflake/BigQuery), it’s fine—just ensure versioning and reproducibility. The anti-pattern is ephemeral, unversioned training extracts.

How do I tie model rollout to business guardrails?

Emit business KPIs to Prometheus (or pull from your analytics system) and build canary rules that halt or roll back when KPIs degrade beyond thresholds. Pair with Istio for traffic ramp and Argo Rollouts/Seldon for orchestrated promotion.

What measurable outcomes should I expect?

Teams who implement this pattern typically see 50–80% reduction in MTTR for data incidents, 99%+ freshness attainment, 30–60% fewer bad model releases (thanks to canaries), and faster retrains (days to hours) due to reproducibility.

Data-engineering · Oct 11, 2025 · 10 minute read

Your Model Didn’t Fail — Your Data Pipeline Did: Training+Serving Data That Doesn’t Lie

The fastest way to fix ML in production isn’t a new model—it’s a reliable data pipeline that feeds training and serving from the same, tested truth.

Alex Porter

Principal Architect, GitPlumbers

20 years building and fixing data systems at scale. Ex-Netflix data platform, ex-FinTech SRE. I’ve broken plenty so you don’t have to.

If your training and serving paths don’t share code, tests, and data contracts, you don’t have one model—you have two that disagree in production.

Back to all posts

Your Model Didn’t Fail — Your Data Pipeline Did: Training+Serving Data That Doesn’t Lie

Key takeaways

Implementation checklist