Databricks Evolves Data Sharing for AI

Databricks is pushing the boundaries of data collaboration with the introduction of OpenSharing, a significant evolution of its Delta Sharing protocol. This move is designed to address the burgeoning needs of the agentic AI era, where sharing extends beyond raw data to include sophisticated AI models and autonomous agents.

Five years ago, Delta Sharing aimed to democratize data access by enabling zero-copy sharing across organizational and platform boundaries. It quickly became a widely adopted standard, facilitating collaboration for major enterprises like SAP and Mercedes-Benz. However, the landscape has shifted dramatically with the rise of AI.

The Agentic Era Demands More Than Just Data

Traditional sharing protocols struggle to accommodate the complexities of AI, which often involve semantic context, AI skills, and unstructured data. These legacy systems are frequently vendor-locked, unable to handle AI logic, and rely on cumbersome networking configurations.

OpenSharing tackles these limitations head-on. It's an independent open-source project, now hosted by the Linux Foundation, that expands the sharing paradigm to encompass the entire AI stack. This includes models, agents, and other AI-driven assets, promising interoperability across any cloud, vendor, or data format.

Matei Zaharia, Co-founder and CTO of Databricks, stated, "Delta Sharing proved the industry would choose open over locked-in. OpenSharing extends that principle to the full AI stack, while expanding the cross-platform ecosystem to Iceberg recipients and on-premises providers. The agentic era deserves an open foundation, and OpenSharing delivers it."

OpenSharing on Databricks: A Two-Layered Approach

Databricks OpenSharing operates on two key levels. The open-source protocol itself provides the foundational specification for vendors and developers to implement. Databricks' enterprise implementation builds upon this, integrating features like Unity Catalog for robust governance and audit logging, and the Databricks Marketplace for discoverability.

Key Features for the AI-Powered Enterprise

Genie Agent Sharing: This groundbreaking feature allows organizations to share governed AI experiences, not just static datasets. Genie Agents, Databricks' AI-powered conversational analytics environments, can now be shared with partners and customers. This includes their underlying semantic context, business metrics, and reusable AI logic. Providers can enforce granular controls, such as restricting data access, setting daily prompt quotas, and capping row export limits, unlocking new monetization models like usage-based pricing.

SecureConnect and Global Distribution: Addressing the persistent challenges of cross-cloud data sharing, OpenSharing on Databricks offers solutions for both networking and cost. SecureConnect simplifies multi-cloud networking by acting as a Databricks-managed proxy, eliminating the need for extensive firewall coordination. Global Distribution combats escalating egress costs through automatic cross-region and cross-cloud replication, allowing recipients to query local replicas with low latency and no egress fees.

Open Client Interoperability & On-prem Storage Ecosystem: True openness means meeting partners where they are. OpenSharing supports formats like Delta Lake, Apache Iceberg, and Parquet, enabling sharing with any Iceberg-compatible client. Furthermore, the Databricks Storage Ecosystem extends the platform's capabilities to on-premises, private cloud, and edge environments. This allows valuable data residing outside the cloud to be accessed and governed without migration or duplication. Launch partners for this initiative include MinIO and Everpure, with more slated to join.

The protocol's architecture ensures that data remains with the provider, with recipients querying live data directly. Unity Catalog provides end-to-end governance, auditing every access and enforcing policies. This approach guarantees a single source of truth and compliance across all shared assets.

The evolution of Delta Sharing into OpenSharing signifies Databricks' commitment to fostering an open, collaborative ecosystem for the next generation of data and AI innovation.