How do I know if I should keep transforms in the warehouse vs. Spark?

If 80% of your transforms are SQL-friendly (aggregations, joins, windows) and your data already lives in Snowflake/BigQuery, keep them there. Move to Spark/Databricks when you need custom UDFs, complex sessionization, or ML that benefits from the lakehouse. Always co-locate compute with data to avoid network egress and shuffles.

What’s the quickest win to cut ETL costs without a rewrite?

Stop full reloads. Implement incremental loads with a watermark and `MERGE`. Pair that with partitioning and predicate pushdown. These two changes usually deliver 30–50% reductions in runtime and scanned bytes.

Won’t more quality checks slow my pipeline down?

Not if you tier them. Put fast, blocking checks on critical tables (uniqueness, not-null, recency). Make heavier profiling or anomaly detection non-blocking and run it out-of-band. Use canary runs on samples before full execution to fail fast.

How do I handle late-arriving data and still stay fast?

Use a sliding window strategy. Reprocess the last N days (e.g., 2–3) with idempotent `MERGE`s. Track watermarks per source and only re-open partitions that need it. This keeps the main flow fast while maintaining accuracy.

What metrics should I report to leadership?

Report SLA attainment (freshness/completeness), runtime on the critical path, cost per million rows processed, and MTTR. Leaders don’t care about engine flame graphs; they care that the dashboard is on time and the bill is predictable.

Data-engineering · Oct 6, 2025 · 10 minute read

Stop Paying for Slow ETL: The Playbook That Cut Our Snowflake Bill 42% and Ended 3 AM Pages

If your nightly ETL slips and your warehouse bill spikes, you don’t need another tool—you need a ruthless optimization pass that protects data quality and business SLAs.

Alex Mercer

Principal Data Engineer, GitPlumbers

20 years of breaking and fixing data systems. Ex-Adobe, ex-Shopify. Led migrations to Snowflake and Databricks, built CDC at scale, and carried the pager for more 3 AM ETLs than he cares to remember.

Fast is only useful if it’s repeatable. Optimize for the happy path and the failure path—both cost money.

Back to all posts

Stop Paying for Slow ETL: The Playbook That Cut Our Snowflake Bill 42% and Ended 3 AM Pages

Key takeaways

Implementation checklist