What’s the fastest way to get meaningful ETL speedups without a rewrite?

Compact small files, enable adaptive execution, and fix one skewed join with a broadcast or bucketing. Add basic dbt/GE checks so you don’t ship garbage faster. You’ll usually see 20–40% wins in a week.

Parquet `zstd` vs `snappy`?

Use `zstd` for batch-heavy tables where CPU is cheaper than I/O; use `snappy` for interactive workloads and frequent read-modify-write. Measure wall time and CPU utilization to decide.

Delta vs Iceberg for optimization features?

Both give you ACID, compaction, and metadata for pruning. Delta has `OPTIMIZE ZORDER` and is great on Databricks; Iceberg is engine-agnostic with solid partition evolution and bin-packing compaction. Choose based on your engine/ecosystem.

How do I avoid double-counting with retries?

Design idempotent writes with deterministic keys and `MERGE`/upsert semantics. Use watermarks and dedupe windows. For CDC, order by `op_ts`/`lsn` and apply tombstones.

Can I get cost reductions on Snowflake/BigQuery without touching ETL code?

Yes: right-size warehouses/slots, auto-suspend aggressively, add clustering/partitioning, and replace hot queries with materialized views. Tag jobs for chargeback to expose runaway spend.

Data-engineering · Nov 4, 2025 · 10 minute read

Stop Paying for Shuffles: The ETL Tune-Up That Cut Runtime 40% and Spend 35%

You don’t need a rewrite. You need to stop moving bytes you don’t need, fix file layouts, and put quality gates where they matter. Here’s the playbook we use to make ETL faster, cheaper, and more reliable—without breaking business logic.

Back to all posts

Stop Paying for Shuffles: The ETL Tune-Up That Cut Runtime 40% and Spend 35%

Key takeaways

Implementation checklist