How do I decide between streaming and micro-batch for incremental loads?

If your freshness SLO is below ~5 minutes and the source supports event streams, use streaming (Kafka, Pub/Sub, Kinesis) with exactly-once semantics and checkpointed state. Otherwise, hourly micro-batches with CDC tables achieve 80–90% of the benefit at a fraction of the complexity. Start with micro-batch; move slices to streaming only where the business case is clear (fraud, pricing, alerts).

When should I still do a full refresh?

Use full refreshes for initial backfills, major model redesigns, or when upstreams are known-bad and you need to re-derive truth from raw logs. Make it explicit: a one-off job with a capped window and a post-refresh snapshot diff to validate parity. Never leave full refreshes as the default path in production DAGs.

What about schema drift from upstream services?

Use explicit data contracts and schema registries. Enforce backward-compatible changes; fail fast on breaking changes at ingestion. In warehouses, prefer permissive landing zones (VARIANT/JSON columns) with controlled projections into typed models, guarded by tests. Add alerts on column count and nullability deltas.

Our codebase is a mess of AI-generated Airflow DAGs and notebooks. Where do we start?

Triage the top 10 jobs by cost and runtime with lineage. Replace notebook glue with versioned jobs. Add incremental strategies, storage pruning, and right-sized compute first. In parallel, layer dbt tests and Great Expectations on business-critical models. We call this a vibe code cleanup — we keep what’s correct, delete the cargo cult, and instrument everything.

Data-engineering · Dec 10, 2025 · 9 minute read

The Nightly ETL That Ate Our Cloud Bill — And the Fix That Cut Runtime 85%

War story: a retail data platform burning money on full refreshes, fixed with incremental loads, storage layout, and right-sized compute — without sacrificing data trust.

Back to all posts

The Nightly ETL That Ate Our Cloud Bill — And the Fix That Cut Runtime 85%

Key takeaways

Implementation checklist