How do I know if I should keep transforms in the warehouse vs. Spark?

If 80% of your transforms are SQL-friendly (aggregations, joins, windows) and your data already lives in Snowflake/BigQuery, keep them there. Move to Spark/Databricks when you need custom UDFs, complex sessionization, or ML that benefits from the lakehouse. Always co-locate compute with data to avoid network egress and shuffles.

What’s the quickest win to cut ETL costs without a rewrite?

Stop full reloads. Implement incremental loads with a watermark and `MERGE`. Pair that with partitioning and predicate pushdown. These two changes usually deliver 30–50% reductions in runtime and scanned bytes.

Won’t more quality checks slow my pipeline down?

Not if you tier them. Put fast, blocking checks on critical tables (uniqueness, not-null, recency). Make heavier profiling or anomaly detection non-blocking and run it out-of-band. Use canary runs on samples before full execution to fail fast.

How do I handle late-arriving data and still stay fast?

Use a sliding window strategy. Reprocess the last N days (e.g., 2–3) with idempotent `MERGE`s. Track watermarks per source and only re-open partitions that need it. This keeps the main flow fast while maintaining accuracy.

What metrics should I report to leadership?

Report SLA attainment (freshness/completeness), runtime on the critical path, cost per million rows processed, and MTTR. Leaders don’t care about engine flame graphs; they care that the dashboard is on time and the bill is predictable.

Data-engineering · Oct 6, 2025 · 10 minute read

Stop Paying for Slow ETL: The Playbook That Cut Our Snowflake Bill 42% and Ended 3 AM Pages

If your nightly ETL slips and your warehouse bill spikes, you don’t need another tool—you need a ruthless optimization pass that protects data quality and business SLAs.

Back to all posts

Stop Paying for Slow ETL: The Playbook That Cut Our Snowflake Bill 42% and Ended 3 AM Pages

Key takeaways

Implementation checklist