Incremental vs CDC: which should I prioritize?

CDC. If you can get reliable change streams (Debezium, Fivetran CDC, native logs), you cut compute and improve correctness. If CDC isn’t available, implement dbt incrementals with `MERGE` and a robust watermark, then backfill occasionally.

We have many small tables—do partitioning and clustering still help?

Yes, but be selective. Partition only where pruning occurs on most queries (e.g., `event_date`). Otherwise, cluster by a join/filter column with good cardinality. Avoid over-partitioning; too many partitions can hurt performance.

Are materialized views worth it in BigQuery/Snowflake?

For stable aggregates queried frequently, yes. They reduce scanned bytes and latency. Watch refresh costs and staleness windows; use them for hot paths, not everything.

How do I prevent cost spikes from backfills?

Throttle with Airflow pools and `max_active_runs`, use smaller dedicated warehouses (Snowflake) or limited slots (BigQuery reservations), and slice backfills by date. Monitor cost/run and stop-on-budget guards.

Delta vs Iceberg—does it matter for cost?

Both can be cheap if you compact and prune. Cost differences usually come from table maintenance discipline and query patterns, not the table format itself.

Data-engineering · Nov 18, 2025 · 10 minute read

The ETL That Ate Your Cloud Bill: How We Cut 68% Runtime and 45% Cost Without Rewriting Everything

Real-world ETL optimization that shrinks runtime and spend while boosting reliability, data quality, and business impact.

Back to all posts

The ETL That Ate Your Cloud Bill: How We Cut 68% Runtime and 45% Cost Without Rewriting Everything

Key takeaways

Implementation checklist