Visual TL;DR
Petabyte Scale IngestDriver
need for massive data ingestion without infrastructure management
From the articleAccording to the Databricks blog post, Zerobus Ingest demonstrated the ability to ingest one petabyte of data in under 24 hours, maintaining a stable throughput of 12 GB/s to a single table during benchmarks.
Databricks Zerobus IngestCore
From the article 6 mentionsDatabricks has launched Zerobus Ingest, a serverless streaming API designed to handle petabyte-scale data pipelines without requiring manual infrastructure setup.
Eliminates InfrastructureContext
no manual setup or management of message queues like Kafka
From the articleDatabricks has launched Zerobus Ingest, a serverless streaming API designed to handle petabyte-scale data pipelines without requiring manual infrastructure setup.
Push-Based APIContext
From the article 2 mentionsThe system bypasses the need for traditional message queues like Kafka, offering a push-based API that accepts data from any producer and writes it to the lakehouse.
Autoscaling MechanismCore
achieved through dynamic partitioning for efficient scaling
From the articleAt the core of Zerobus Ingest's capability is its autoscaling mechanism, achieved through dynamic partitioning.
Unity CatalogContext
governs time-series data ingested into Delta tables
From the articleThis new service promises to ingest massive volumes of time-series data from sources like IoT sensors and autonomous vehicles directly into Delta tables, governed by Unity Catalog.
1 PB in < 24 HoursOutcome
From the articleAccording to the Databricks blog post, Zerobus Ingest demonstrated the ability to ingest one petabyte of data in under 24 hours, maintaining a stable throughput of 12 GB/s to a single table during benchmarks.
12 GB/s ThroughputOutcome
stable ingest rate achieved per single table
From the article 2 mentionsThis component parses data efficiently without unnecessary memory allocations, achieving high throughput even with dynamic schemas.
