Databricks says most teams should stop adding Flink just for speed. Spark Real-Time Mode now hits millisecond latency on the same engine used for batch, SQL and machine learning.
I buy that default.
Databricks tested customer-derived workloads at 10,000 records per second for 30 minutes. It measured Kafka input to output latency with Spark on Databricks against Flink on AWS Managed Service for Apache Flink 1.18. Stateless work tied at 3 ms p99, stateful aggregation was 14 ms for Spark against 45 ms for Flink, and stream-static joins were 9 ms against 95 ms, according to Databricks, which compared its worst Spark run to its best Flink run and called the gap up to 92% faster.
Production cases point the same way. MakeMyTrip considered Flink, chose Spark Real-Time Mode to avoid a second engine, and cut P50 to 44 ms and P99 from more than a minute to about 500 ms, according to Databricks. Coinbase and DraftKings also run Real-Time Mode in production, according to Databricks. The move is a trigger change on an existing Structured Streaming query, with stateful support in open source Spark 4.3 and generally available on Databricks on classic compute or in Lakeflow Spark Declarative Pipelines.
The declarative push is wider than Databricks. Materialized lake views in Fabric let teams define Bronze, Silver and Gold layers as declarative SQL or Python instead of hand-written jobs, per Brolly Academy. That is the same trade as declarative pipelines over wired jobs.
Flink still has a fair case in narrow spots. Very large keyed state beyond local disk, pattern matching with CEP or SQL MATCH_RECOGNIZE, heavy event-time timers that Real-Time Mode does not yet support in transformWithState, or a Flink platform already in production can justify staying, according to Databricks.
That exception leaves the rule intact for new work. If you are starting today, start on Spark and make Flink prove it is needed before you operate it.
