How do I choose between Flink and Spark Structured Streaming?

If you need low-latency event-time joins, complex state, exactly-once sinks, and predictable backpressure handling, Flink is the safer bet. Spark can work for micro-batch patterns and teams already deep on Spark, but watch state store size and checkpoint commit latency. We use Flink for hot paths and keep Spark for batch/ML training and serving Iceberg tables.

Can I get exactly-once with Kafka and Iceberg?

Yes. Use Kafka idempotent producers and transactions on the write path. In Flink, enable checkpointing and use transactional sinks for Kafka and Iceberg (Iceberg’s Flink sink commits atomically with snapshot isolation). Ensure your sinks support two-phase commits and your checkpoint storage is reliable (S3/GCS/ADLS), with timeouts tuned to your throughput.

How much retention do I need on Kafka topics?

Work backward from replay goals. If you need to reprocess 24 hours at 3x speed, keep at least 48 hours hot and push older segments to tiered storage. For compacted topics, size for compaction lag and tombstones. Monitor storage cost vs replay RTO and adjust; most teams under-retain until their first backfill fire drill.

Do I need a Schema Registry if I’m using JSON?

Yes — or stop using JSON for critical paths. Schema Registry with Avro/Protobuf gives you compatibility rules and evolution controls. JSON without a registry devolves into breaking changes at 2 a.m. If you must do JSON, use JSON Schema in a registry and enforce compatibility at producers.

What’s the fastest way to add quality checks without slowing down the pipeline?

Validate at the edge with lightweight checks (required fields, type, ranges) and route failures to a DLQ. Run deeper statistical checks (drift, anomalies) asynchronously off the DLQ or Iceberg tables. Keep hot-path logic deterministic and idempotent; do heavy lifts out-of-band.

Data-engineering · Dec 11, 2025 · 10 minute read

The “Real-Time” Pipeline That Stalled at Lunch — And How We Stopped Losing Money by 12:05

Streaming data isn’t about speed; it’s about reliability under pressure. Here’s how we design Kafka/Flink/Iceberg stacks that absorb spikes, preserve quality, and ship decisions that move revenue.

Back to all posts

The “Real-Time” Pipeline That Stalled at Lunch — And How We Stopped Losing Money by 12:05

Key takeaways

Implementation checklist