Spark Streams Tackle Fraud in Milliseconds

Financial institutions are racing to stop fraud before it happens, a challenge complicated by the sheer speed of digital transactions. Traditional methods often involve slow batch processing or bolting on separate streaming engines, leading to operational complexity and delayed detection. Databricks aims to simplify this with its new solution, combining Spark Real-Time Mode and Lakebase for end-to-end fraud detection on a single platform.

The core problem is speed versus simplicity. Fraudsters can exploit stolen card details in seconds, making real-time intervention critical. However, building and managing separate streaming infrastructure alongside existing data platforms creates duplicated systems, split governance, and increased engineering burden. This dual system approach historically forced a choice between speed and operational ease.

Spark Real-Time Mode: Sub-Second Processing Without the Overhead

Spark Real-Time Mode (RTM) is an evolution of Spark Structured Streaming designed for latency-sensitive applications. It achieves sub-300ms stream processing, reportedly outperforming Apache Flink in key workloads and enabling companies like Coinbase to compute hundreds of ML features with sub-100ms latency. Crucially, RTM operates within the existing Spark engine, eliminating the need for separate streaming stacks. This unification prevents logic drift, as the same code used for offline training can be applied to real-time scoring. It also consolidates operational tooling and reduces on-call responsibilities.

This technology is a significant step towards achieving low-latency processing, as discussed in our coverage of Spark Streaming hitting millisecond latency.

Lakebase: Integrated Postgres for Low-Latency Serving

The solution also leverages Databricks Lakebase, a fully managed, serverless PostgreSQL database embedded within the Databricks platform. Lakebase acts as a low-latency serving layer for feature enrichment, providing context from merchant risk profiles and cardholder data. This avoids the latency typically associated with broadcast joins in streaming pipelines.

The architecture, demonstrated through a credit card transaction scenario, ingests data from Kafka, processes it using Spark RTM for parsing, velocity tracking, enrichment, and scoring. Decisions are then routed to approve, flag, or block transactions. End-to-end latency tests show P99 performance between 215-392ms, validating production readiness without external infrastructure.

Upgrading to Machine Learning

Beyond static rules, the solution integrates machine learning models. This upgrade allows for reduced false positives and adaptation to evolving fraud patterns. MLflow's experiment tracking and versioning provide necessary model lineage for regulatory compliance. Lakebase is continuously updated with per-card features, enabling dynamic model scoring.