Databricks Touts Unity Catalog for Iceberg

Databricks is doubling down on open data formats with significant advancements for Apache Iceberg™ within its Unity Catalog. The company announced general availability for Managed Iceberg, Iceberg v3, and Foreign Iceberg, positioning Unity Catalog as a comprehensive and production-ready solution for the open lakehouse. This move signals a push towards greater interoperability and governance for data across diverse engines.

The platform now boasts five core capabilities designed to set it apart: open APIs for engine flexibility, catalog federation for a unified view across disparate data estates, cross-engine access control for consistent governance, zero-copy secure sharing, and AI-driven optimization for performance. These features aim to address the growing need for governed data access across an expanding ecosystem of AI and agentic applications.

Iceberg Capabilities Go GA

The latest updates bring a suite of Iceberg features to general availability and preview. Managed Iceberg allows users to create, read, write, and optimize Iceberg tables directly within Unity Catalog, with features like Predictive Optimization and Liquid Clustering automating performance tuning. Iceberg v3 support is now native, incorporating deletion vectors, row tracking, and the VARIANT type across managed and foreign tables. Foreign Iceberg support is also GA, enabling governance and querying of Iceberg tables managed externally. Databricks is also pushing forward with Iceberg-compatible materialized views and cross-engine Attribute-Based Access Control (ABAC) in beta.

Catalog federation is expanding, with new connectors for Google Cloud Lakehouse and Palantir, adding to existing integrations with AWS Glue, Snowflake Horizon, Hive Metastore, and Salesforce Data Cloud. This aims to make Unity Catalog the central management layer for an organization's entire Iceberg data estate.

Five Pillars of Interoperability

Databricks argues that a truly open lakehouse catalog must offer more than basic metadata tracking. Unity Catalog addresses five fundamental requirements: open APIs, federation, cross-engine governance, secure sharing, and continuous performance innovation.

Open APIs and credential vending allow any Iceberg-compatible client, from Spark to DuckDB, to interact with tables in Unity Catalog without data duplication or broad storage permissions. The platform also vends credentials for federated Iceberg tables, enhancing secure access to externally managed data.

Catalog federation provides a singular view across multiple catalogs like AWS Glue and Snowflake Horizon, allowing users to govern and query external Iceberg tables directly. This unified approach simplifies management for complex, distributed data environments.

Cross-engine ABAC, currently in beta, extends fine-grained governance policies. Administrators define policies once in Unity Catalog, which are then enforced by external Iceberg engines via the Iceberg REST Catalog Scan APIs, ensuring consistent access control regardless of the query engine.

Zero-copy secure sharing is enhanced through Delta Sharing, now fully supporting Iceberg as both a source and destination format. This enables secure, live data sharing with any Iceberg REST-compatible client without requiring data ingestion or copies. Foreign Iceberg sharing is also in public preview, allowing governed sharing of externally managed Iceberg tables.

Performance and format innovation is driven by AI. Predictive Optimization automatically tunes table performance based on workload patterns, benefiting all engines accessing the data. The integration of Iceberg v3 features like deletion vectors and row tracking aims to close performance gaps between Delta and Iceberg, enabling interoperability without data rewrites.

The company sees a future where open lakehouse catalogs like Unity Catalog will be crucial for governing data across systems, especially as AI and agentic applications become more prevalent. The convergence of Iceberg v4 and Delta 5.0 on a unified metadata structure is anticipated to resolve the long-standing trade-off between interoperability and production-ready performance, according to Databricks.