Iceberg REST: Snowflake vs. Databricks

The promise of open interoperability in data platforms hinges on seamless data access across different tools and engines. This means securely accessing governed data through published, standards-based interfaces, breaking free from proprietary lock-in. It empowers organizations to choose the best tools for each workload without costly data migrations.

This principle is central to how Snowflake approaches Apache Iceberg™. The goal is to allow customers to use their Iceberg tables on Snowflake or elsewhere, with full read and write capabilities, without forcing migration to a specific catalog. This is where the concept of Bidirectional Iceberg REST becomes critical.

The Interoperability Divide

True interoperability demands both inbound and outbound data federation using open standards. Inbound federation allows external engines to read and write to your Iceberg tables. Outbound federation means your systems can interact with Iceberg tables managed by other catalogs.

Databricks Unity Catalog, while claiming interoperability, falls short. Currently, it only supports inbound access to its Iceberg tables. For outbound access, Databricks uses a proprietary SDK via JDBC to read table metadata, and crucially, this is read-only. Writes to external Iceberg tables managed by other catalogs are not supported.

This limitation forces users to migrate their entire data estate to Unity Catalog to achieve full functionality, undermining the very concept of open interoperability. It creates a vendor lock-in scenario, contrary to the stated goals.

Snowflake's Horizon Catalog: A Different Approach

In contrast, Snowflake Horizon Catalog offers a complete implementation of the Iceberg REST Catalog protocol. This native governance layer provides consistent management of classification, access policies, lineage, and quality across all Iceberg tables, whether managed by Snowflake or accessed externally.

Horizon connects to external catalogs like Unity Catalog via Catalog Linked Databases, directly calling their Iceberg REST Catalog endpoints. This eliminates the need for complex credential management or security workarounds.

The Importance of Open Security Standards

The Iceberg REST Catalog specification includes a feature called vended credentials. These are short-lived tokens delegated by the source catalog for specific operations, ensuring granular security. Without this, systems often resort to broad IAM roles and long-lived credentials, increasing security risks and complexity.

Snowflake Horizon Catalog fully implements vended credentials for both inbound and outbound federation. This allows access policies defined in Horizon to be enforced across different engines without duplication. The new Scan Plan API further extends this to fine-grained row and column level access controls.

Databricks Unity Catalog, however, does not consume credentials vended by external catalogs, failing to respect source catalog access policies.

Databricks Compute Without Unity Catalog

Interestingly, Databricks compute, when configured without Unity Catalog, can achieve full read/write capabilities with external Iceberg REST-compliant catalogs like Snowflake Horizon. This suggests the limitations are specific to Unity Catalog's integration model, not the broader Databricks platform.

Achieving true data agency requires freedom of choice and the ability to use preferred tools without forced migrations or siloed security. Prioritizing catalogs that implement open security and bidirectional iceberg rest is key to ensuring data remains a strategic, controllable asset.