Databricks Unifies Lakehouse with Managed Tables

Databricks is pushing its Lakehouse vision forward with a significant update to its Unity Catalog managed tables, now allowing external engines like Apache Spark, Flink, and DuckDB to create, read, and write data directly. This move aims to enhance interoperability, boost performance, and solidify unified governance across diverse data workloads.

Previously, achieving multi-engine access often meant relying on external tables, which lacked Databricks' built-in performance optimizations and strict governance guarantees. This update removes that trade-off, positioning managed tables as the clear choice for price, performance, and ecosystem flexibility.

Unified Governance and Performance Boosts

At the core of the enhancement is Unity Catalog's role as the central governance layer. Access policies are enforced consistently, regardless of the engine used. This centralized control is crucial for complex, multi-engine pipelines where data might be ingested via streaming, transformed by Spark, and then queried by tools like Starburst or DuckDB.

Managed tables also leverage Databricks' Predictive Optimization. This feature automatically tunes table layouts, collects query statistics, and evolves clustering to improve query speeds by up to 20x and reduce storage costs by half, all without manual intervention. This capability is foundational for features like disaster recovery and Zerobus ingestion.

Expanding the Ecosystem with Open Standards

External engine access is built upon open APIs from the open-source Unity Catalog (UC OSS) project. This ensures compatibility not only with Databricks' Unity Catalog but also with self-hosted UC OSS deployments. This openness fuels a growing ecosystem, with partners like Starburst already integrating to support read and write operations.

Databricks is further investing in the Delta Lake and Unity Catalog OSS ecosystem through Delta Kernel. This provides libraries for interacting with Delta tables, enabling engines to integrate seamlessly with managed catalogs. DuckDB, for instance, has built extensions on Delta Kernel, allowing direct interaction with managed Delta tables.

This initiative aligns Delta Lake with the catalog-managed model of Iceberg, offering catalog benefits while preserving broad engine interoperability. This is a key step in Databricks' ongoing efforts to enhance Databricks Lakehouse interoperability and expand its Databricks Lakehouse interoperability.

Seamless Upgrades and Future Access

Existing external tables can be upgraded to managed tables with a simple ALTER TABLE SET MANAGED command, preserving interoperability while gaining Predictive Optimization and catalog commit benefits. This is part of Databricks' broader push for simplified data management, similar to its Unity Catalog managed tables.

The public preview for external access to UC managed Delta tables is now available, supporting create, read, and write operations. Users can enable external data access on their Unity Catalog metastore and grant necessary privileges to external engines.