What latency qualifies as “real-time” for decisioning?

Define it per use case. For inventory and fraud ops, we commonly set 95% < 30s end-to-end freshness and 99% < 120s. For ad bidding, you need sub-second and a different architecture. Don’t ship a number without an SLO.

Do I need Flink, or can I use Spark Structured Streaming?

Both work. If you need low-latency, long-lived state and exactly-once with Kafka transactions, Flink is usually simpler. If you’re already on Databricks and focus on Delta + batch/stream unification, Spark is pragmatic.

How do I prevent breaking changes to schemas?

Put Protobuf/Avro in Git, enforce BACKWARD compatibility in CI, and block merges on failures. Maintain consumer-driven contracts and publish deprecation timelines. Use Schema Registry compatibility settings per subject.

What’s the first thing to do if I inherit an AI-generated pipeline?

Run a vibe code cleanup: add checkpointing, backpressure handling, metrics, and tests. Strip println debugging, lock down schemas, and instrument with Prometheus. We do quick code rescue sprints for exactly this.

Data-engineering · Nov 27, 2025 · 9 minute read

The Real-Time Data Pipeline That Actually Drives Decisions (Not Dashboards)

Q: Do I need Flink, or can I use Spark Structured Streaming?

Both work. If you need low-latency, long-lived state and exactly-once with Kafka transactions, Flink is usually simpler. If you’re already on Databricks and focus on Delta + batch/stream unification, Spark is pragmatic.

Q: How do I prevent breaking changes to schemas?

Put Protobuf/Avro in Git, enforce BACKWARD compatibility in CI, and block merges on failures. Maintain consumer-driven contracts and publish deprecation timelines. Use Schema Registry compatibility settings per subject.

Q: What’s the first thing to do if I inherit an AI-generated pipeline?

Run a vibe code cleanup: add checkpointing, backpressure handling, metrics, and tests. Strip println debugging, lock down schemas, and instrument with Prometheus. We do quick code rescue sprints for exactly this.

How to build reliable, observable, and evolvable streams executives trust when the money is on the line.

Back to all posts

The Real-Time Data Pipeline That Actually Drives Decisions (Not Dashboards)

Key takeaways

Implementation checklist