Do I need a feature store, or can I roll my own with tables and Redis?

You can roll your own, and that’s how most teams end up with skew. A feature store like Feast gives you offline/online parity, point-in-time joins, and a single definition of features. That’s the difference between “works in training” and “works in prod.”

What’s the minimum viable stack to get reliability?

Contracts + tests + versioning. Specifically: Schema Registry on ingestion topics, dbt tests + Great Expectations on critical models, and versioned data (Delta/Iceberg) with MLflow run tracking. You can add KServe/Istio and Feast as you scale.

How do I quantify training-serving skew?

Track distributional distance (e.g., Jensen–Shannon divergence) for key features between offline and online over a rolling window. Alert when it exceeds a threshold (e.g., 0.02). Tie breaches to automated rollback for canaries.

We’re on Snowflake/BigQuery—does this still apply?

Yes. Swap Spark/Delta for Snowflake/BigQuery-native transforms. Feast supports both as offline stores. Use Redis/DynamoDB for online. KServe works regardless of your warehouse; the key is consistent feature definitions and versioned snapshots.

Isn’t this overkill for a small team?

It’s cheaper than firefighting. Start with the top 5 data sources and 1–2 feature groups. The contracts + tests + versioning combo removes 80% of surprises. You can layer on serving discipline as traffic grows.

Data-engineering · Oct 4, 2025 · 9 minute read

Stop Blaming the Model: Build ML Data Pipelines That Don’t Lie in Training or Serving

What it takes to ship reliable ML: contracts, quality gates, feature parity, and canary serving—backed by metrics, not vibes.

Alex Ramirez

Principal Data & ML Engineer, GitPlumbers

20 years building and fixing data systems at scale. Ex-Netflix data platform, led ML infra at a fintech unicorn, and has battle scars from every broken Airflow DAG you can imagine.

If it’s not enforced in automation, it’s not a contract—it’s a suggestion.

Back to all posts

Stop Blaming the Model: Build ML Data Pipelines That Don’t Lie in Training or Serving

Key takeaways

Implementation checklist