How “real-time” do I actually need?

Start from the decision. If a human action or automated control loop needs an update within 2 minutes to avoid loss or capture upside, set p99 end-to-end latency at 120s and back into partitioning, parallelism, and windowing that meet it. For fraud or ads, 1–5 seconds might be warranted. Document degraded modes if you miss it.

Do I really need Flink?

If you have stateful joins, exactly-once sinks, and watermark logic, yes — Flink or ksqlDB makes it maintainable. If it’s simple ETL with low state, Kafka Streams can be fine. Don’t write a bespoke consumer unless you’re ready to rebuild state, checkpointing, and backpressure handling.

Why not just use my data warehouse for everything?

Warehouses (BigQuery/Snowflake) are great for analytics, not sub-second operational decisions. You can land streams there for broader analysis, but your real-time control surfaces want a low-latency OLAP store like ClickHouse/Pinot with materialized views and streaming sinks.

How do I handle schema evolution without breaking consumers?

Data contracts + schema registry with strict compatibility. Use additive changes, version topics for breaking changes (v1 → v2), and run a canary with output diffing. Communicate deprecation dates like an API; don’t hot-swap types in place.

Data-engineering · Oct 4, 2025 · 10 minute read

Real-Time Data Pipelines That Don’t Lie: Decisions You Can Bet the Quarter On

If your “real-time” dashboard is 12 minutes stale, you don’t have a pipeline — you have a liability. Here’s the architecture, guardrails, and roll-out plan we use to ship reliable, business-critical streams.

Alex Mercer

Principal Engineer, GitPlumbers

20 years building and rescuing data systems at scale. Led streaming platforms at a unicorn marketplace, helped banks survive schema migrations, and made more broken Kafka clusters boring than he can count.

Real-time isn’t about speed; it’s about trust. If the stream isn’t trusted, decisions degrade and the pager lights up.

Back to all posts

Real-Time Data Pipelines That Don’t Lie: Decisions You Can Bet the Quarter On

Key takeaways

Implementation checklist