Databricks Hits Petabyte Scale Ingest

Databricks has launched Zerobus Ingest, a serverless streaming API designed to handle petabyte-scale data pipelines without requiring manual infrastructure setup. This new service promises to ingest massive volumes of time-series data from sources like IoT sensors and autonomous vehicles directly into Delta tables, governed by Unity Catalog.

The system bypasses the need for traditional message queues like Kafka, offering a push-based API that accepts data from any producer and writes it to the lakehouse. According to the Databricks blog post, Zerobus Ingest demonstrated the ability to ingest one petabyte of data in under 24 hours, maintaining a stable throughput of 12 GB/s to a single table during benchmarks.

Architectural Innovations

At the core of Zerobus Ingest's capability is its autoscaling mechanism, achieved through dynamic partitioning. Unlike traditional streaming architectures that require pre-provisioning and managing static partitions, Zerobus shifts the unit of ordering from partitions to stream connections. This allows the system to dynamically scale compute resources up or down based on real-time demand.

This approach ensures that pods can be added during ingestion spikes and removed when demand subsides, leading to efficient compute utilization. The system also incorporates a custom, zero-copy protobuf decoder called ZeroParser. This component parses data efficiently without unnecessary memory allocations, achieving high throughput even with dynamic schemas.

Furthermore, Zerobus Ingest implements a latency-optimized write-ahead log (WAL) to ensure data durability and enable quick message handoff. This WAL, combined with gRPC bidirectional streaming, allows clients to receive acknowledgments for committed data offsets, enabling them to safely clear their in-flight buffers.

Benchmarking with Cosmic Data

To prove its capabilities, Databricks used NASA’s NEOWISE dataset, comprising 200 billion data points over 11 years, for its benchmarks. The test involved simulating a real-world fan-in pattern using Locust to coordinate thousands of concurrent streams, stressing the ingestion service at scale.

The results underscore Zerobus Ingest's ability to manage extreme data volumes and fluctuating ingestion patterns seamlessly. This advancement in Databricks Zerobus Ingest significantly simplifies the process of building and maintaining high-throughput, petabyte-scale streaming data pipelines, streamlining real-time data applications.