Databricks, BigQuery Unite Data Catalogs

Databricks and Google Cloud are bridging their data platforms, enabling bidirectional catalog federation between Databricks Unity Catalog and Google Cloud's Lakehouse. This move aims to eliminate data duplication and streamline governance for shared data assets.

The integration allows users in BigQuery to read tables managed by Unity Catalog, and conversely, Databricks users can access Iceberg tables written from BigQuery. This interoperability is built on open standards, aligning with the growing adoption of formats like Delta Lake and Apache Iceberg. It represents a significant step towards open standards data interoperability, building on previous efforts like Databricks Embraces Iceberg v3.

Google Cloud is introducing catalog federation to Unity Catalog in preview, allowing engines like BigQuery to access Unity Catalog-managed tables without copying data. Databricks, in turn, is launching a private preview for Google Cloud's Lakehouse federation, enabling Unity Catalog to govern and read foreign Iceberg tables managed within Google Cloud.

This capability allows Databricks customers using Google Cloud to seamlessly mount foreign Iceberg tables into Unity Catalog. Unified governance is a key benefit, with Unity Catalog's policies, access controls, and lineage tracking extending to federated data. This means users can query data across their entire estate, secured and contextualized by Unity Catalog, enabling tools like Genie for natural language queries. This interoperability allows for accessing data without duplication, a crucial aspect for modern data stacks, as highlighted in discussions around dbt on Databricks: Open Platform Advantage.

Databricks views this collaboration with Google Cloud as a move towards an ecosystem where innovation and interoperability coexist, rather than being mutually exclusive choices.