# Databricks Hits Petabyte Scale Ingest
_Databricks Zerobus Ingest achieves petabyte-scale data ingestion at 12 GB/s per table, eliminating infrastructure management._
**Published:** 2026-06-11
**Source:** https://www.startuphub.ai/ai-news/technology/2026/databricks-hits-petabyte-scale-ingest
---
Databricks has launched Zerobus Ingest, a serverless streaming API designed to handle petabyte-scale data pipelines without requiring manual infrastructure setup. This new service promises to ingest massive volumes of time-series data from sources like IoT sensors and autonomous vehicles directly into Delta tables, governed by Unity Catalog.

Petabyte Scale IngestDriver

need for massive data ingestion without infrastructure managementFrom the articleAccording to the Databricks blog post, Zerobus Ingest demonstrated the ability to ingest one petabyte of data in under 24 hours, maintaining a stable throughput of 12 GB/s to a single table during benchmarks.Databricks Zerobus IngestCoreFrom the article 6 mentionsDatabricks has launched Zerobus Ingest, a serverless streaming API designed to handle petabyte-scale data pipelines without requiring manual infrastructure setup.Eliminates InfrastructureContextno manual setup or management of message queues like KafkaFrom the articleDatabricks has launched Zerobus Ingest, a serverless streaming API designed to handle petabyte-scale data pipelines without requiring manual infrastructure setup.Push-Based APIContextFrom the article 2 mentionsThe system bypasses the need for traditional message queues like Kafka, offering a push-based API that accepts data from any producer and writes it to the lakehouse.Autoscaling MechanismCoreachieved through dynamic partitioning for efficient scalingFrom the articleAt the core of Zerobus Ingest's capability is its autoscaling mechanism, achieved through dynamic partitioning.Unity CatalogContextgoverns time-series data ingested into Delta tablesFrom the articleThis new service promises to ingest massive volumes of time-series data from sources like IoT sensors and autonomous vehicles directly into Delta tables, governed by Unity Catalog.1 PB in OutcomeFrom the articleAccording to the Databricks blog post, Zerobus Ingest demonstrated the ability to ingest one petabyte of data in under 24 hours, maintaining a stable throughput of 12 GB/s to a single table during benchmarks.with12 GB/s ThroughputOutcomestable ingest rate achieved per single tableFrom the article 2 mentionsThis component parses data efficiently without unnecessary memory allocations, achieving high throughput even with dynamic schemas.
The system bypasses the need for traditional message queues like Kafka, offering a push-based API that accepts data from any producer and writes it to the lakehouse. According to the [Databricks blog post](https://www.databricks.com/blog/ingesting-milky-way-petabyte-scale-zerobus-ingest), Zerobus Ingest demonstrated the ability to ingest one petabyte of data in under 24 hours, maintaining a stable throughput of 12 GB/s to a single table during benchmarks.

## Architectural Innovations

At the core of Zerobus Ingest's capability is its autoscaling mechanism, achieved through dynamic partitioning. Unlike traditional streaming architectures that require pre-provisioning and managing static partitions, Zerobus shifts the unit of ordering from partitions to stream connections. This allows the system to dynamically scale compute resources up or down based on real-time demand.

This approach ensures that pods can be added during ingestion spikes and removed when demand subsides, leading to efficient compute utilization. The system also incorporates a custom, zero-copy protobuf decoder called ZeroParser. This component parses data efficiently without unnecessary memory allocations, achieving high throughput even with dynamic schemas.

Furthermore, Zerobus Ingest implements a latency-optimized write-ahead log (WAL) to ensure data durability and enable quick message handoff. This WAL, combined with gRPC bidirectional streaming, allows clients to receive acknowledgments for committed data offsets, enabling them to safely clear their in-flight buffers.

## Benchmarking with Cosmic Data

To prove its capabilities, Databricks used NASA’s NEOWISE dataset, comprising 200 billion data points over 11 years, for its benchmarks. The test involved simulating a real-world fan-in pattern using Locust to coordinate thousands of concurrent streams, stressing the ingestion service at scale.

The results underscore Zerobus Ingest's ability to manage extreme data volumes and fluctuating ingestion patterns seamlessly. This advancement in [Databricks Zerobus Ingest](/ai-news/technology/2026/databricks-streamlines-real-time-data-apps) significantly simplifies the process of building and maintaining high-throughput, petabyte-scale streaming data pipelines, streamlining real-time data applications.
---
Original analysis from [startuphub.ai](https://www.startuphub.ai), the #1 AI startup directory.