Do I really need exactly-once everywhere?

No. It’s expensive and brittle at boundaries. Use exactly-once (Kafka Streams `exactly_once_v2`, Flink checkpoints) for financial transfers, inventory, and billing. Everywhere else, at-least-once with idempotent upserts and small dedupe windows is cheaper and good enough.

How many Kafka partitions should I create?

Start from consumer parallelism and target throughput. If a consumer instance can process 5k msgs/sec and you need 100k msgs/sec with headroom, 24–32 partitions is reasonable. Avoid hundreds of partitions by default—controller churn and rebalances will tax you.

Where should I validate data: producer or consumer?

Both, but differently. Producers enforce the data contract (shape/types); consumers enforce business rules (ranges, referential integrity). Put light checks at the edge to fail fast, and heavier checks in stream processors with quarantine paths.

Can I keep the warehouse as my source of truth?

Keep the warehouse for analytics, but your source of truth for facts is the event log + bronze tables. That’s what you replay. Warehouses are for aggregates and history, not the canonical event timeline.

Data-engineering · Oct 26, 2025 · 9 minute read

200k msgs/sec Without the Lies: Streaming Data That Stays Clean, Fast, and Auditable

Throughput is cheap. Trust is expensive. Here’s how to design streaming pipelines that don’t gaslight the business when the firehose opens.

Back to all posts

200k msgs/sec Without the Lies: Streaming Data That Stays Clean, Fast, and Auditable

Key takeaways

Implementation checklist