Iceberg or Delta—does it matter?

Both work. If you need multi-engine portability (Spark, Trino, Athena, Snowflake External Tables), Iceberg edges out. If you’re standardized on Databricks with Unity Catalog, Delta is fine. Pick one and commit to its compaction and governance patterns.

Not everywhere. Use CDC streaming into bronze for operational sources where freshness matters. Batch is fine for slow-moving data (ERP, SFTP). The key is consistent contracts and quality checks.

Will this replace my warehouse?

Not necessarily. Many teams run a lake for raw+transforms and push curated marts to Snowflake/BigQuery. Lakes shine for flexibility and ML; warehouses still win for governed BI at scale. Hybrid is normal.

How do we prevent the small files problem?

Control writer batch sizes, set target file size, compact on a schedule, and avoid over-partitioning. Iceberg/Delta compaction should be a first-class Airflow/Dagster job with alerts.

What’s the fastest path to value?

Pick one business KPI, wire CDC to bronze, build a single silver model with dbt tests, then a gold mart with freshness SLOs. Put a dashboard in front of a real stakeholder within 30–45 days.

Data-engineering · Oct 8, 2025 · 10 minute read

Your S3 Isn’t a Data Lake: The Architecture That Survives 10x Growth Without Melting Down

I’ve watched “just dump it in S3” turn into million-dollar outages. Here’s the lake architecture that scales, stays reliable, and actually delivers business value.

Back to all posts

Your S3 Isn’t a Data Lake: The Architecture That Survives 10x Growth Without Melting Down

Key takeaways

Implementation checklist