Databricks released UCDataset in Beta on AI Runtime. The native PyTorch loader streams Unity Catalog tables directly to GPUs and cut training time by 10x in testing.
No exports. No copies.
GPU bills drive the timing. Teams export tables to Parquet before each run. They store a second copy. It goes stale fast and sits outside lineage, the company said.
UCDataset reads table files directly from cloud storage, the company said. It splits matching rows evenly across every GPU and data loader worker. Each chip reads only its share.
The benchmark used a 300 GB feature table across 8 NVIDIA H100 GPUs to train a recommendation model. The 10x improvement was measured against loading an exported Parquet copy of the same data, the company said.
ML engineers describe what to read. They pick the table, columns and row filter. The filter applies during the read, so unmatched rows never reach training code.
The example is main.recsys.ctr_features for click-through-rate training on 16 GPUs across two nodes. New impressions land daily. The next run picks up appended rows automatically, with deletion vectors applied.
Governance carries over. Jobs read as the identity that runs them. Unity Catalog permissions apply exactly as elsewhere, the company said. Install is %pip install databricks-uc-dataset with standard PyTorch DDP and torchrun.
Beta reads Delta tables in Unity Catalog. It cannot read tables with row filters or column masks. For files in volumes, Databricks points users to UCVolumeDataset, with FILE type support planned for images, audio and documents.
Public benchmarks live elsewhere. The UCI Machine Learning Repository is a public archive hosted at archive.ics.uci.edu by the University of California, Irvine, per a CASRAI guide. UCDataset targets private production tables that must stay governed. Not financial advice. Markets move fast.
