Do we really need streaming, or is micro-batch “right-time” enough?

If your decisions can tolerate 5–15 minutes of delay (e.g., BI dashboards), micro-batch with aggressive scheduling may be fine. But if you’re driving pricing, fraud prevention, inventory allocation, or SLA credits, you need event-driven CDC, watermarks, and low-latency error handling. We often start with micro-batch during shadowing, then promote hot paths to Flink once the SLO delta is clear.

How do you guarantee exactly-once?

You don’t, universally. You engineer for idempotency. In practice: Kafka idempotent producers + transactions where possible; Flink checkpointed state and transactional sinks (Iceberg/Snowflake MERGE with deterministic keys); dedup on keys + event-time ordering with watermarks. Then prove it with chaos testing and data-diff.

Snowflake vs Iceberg/Delta for the curated layer?

If your org is Snowflake-first, use Snowpipe Streaming and MERGE for upserts, but keep Kafka retention for replay. If you want open formats and cheap long-term storage, Iceberg/Delta on S3/GCS with Flink writes is excellent. We deploy hybrids: stream to Iceberg for bronze/silver, expose gold marts in Snowflake/BigQuery for analysts.

Our pipelines were partially generated by an LLM. Is that a problem?

Not inherently. It’s a problem when it ships without contracts, tests, and lineage. We do AI code refactoring and vibe code cleanup: add schema contracts, GE/dbt tests, SLO instrumentation, and runbooks. Treat AI output like a junior engineer’s draft — useful, but verify.

What’s the fastest path from zero to a decision-grade MVP?

30–45 days if you’ve got cloud access: Terraform a managed Kafka (Confluent/Redpanda), Debezium for 1–2 critical tables, Flink SQL for the hot path, Iceberg on S3, Great Expectations gate, dbt model, Airflow/Dagster with OpenLineage, Prometheus alerts. Shadow for two weeks, then canary. Keep scope brutally narrow.

Data-engineering · Nov 30, 2025 · 10 minute read

Real-Time Pipelines That Don’t Lie: Shipping Decision‑Grade Data Under SLA, Not Vibes

If your CFO is making calls off streaming dashboards, you can’t afford “eventual” truth. Here’s the blueprint we deploy when reliability, quality, and minutes‑level latency actually matter.

Back to all posts

Real-Time Pipelines That Don’t Lie: Shipping Decision‑Grade Data Under SLA, Not Vibes

Key takeaways

Implementation checklist