Databricks Unifies Unstructured Data for AI

Databricks is making it easier to bring unstructured data like documents, images, and video into the governed AI workflows of its Lakehouse platform. The company announced the beta availability of its new FILE type, a native column format designed to store and manage multimodal data directly alongside structured tables. This move aims to eliminate data silos and simplify the process of building AI applications that require diverse data formats.

Traditionally, unstructured data has lived in separate systems, creating governance, security, and accessibility challenges. The new FILE type, introduced in a blog post by Databricks engineers, treats these files as first-class citizens within the Lakehouse. This means applying the same fine-grained access controls and security policies used for structured data to raw files. Automated compliance features, such as deleting a file when its corresponding row is removed, also address regulatory needs like GDPR's 'right to be forgotten'.

Beyond Structured Data

The vast majority of an organization's data estate often lies in unstructured formats. Think of contracts, product images, customer call recordings, or dashcam footage. While AI has shown immense potential in extracting value from this data, managing and querying it alongside traditional databases has been a significant hurdle. Databricks’ FILE type directly addresses this by storing lightweight pointers to files rather than the binaries themselves, processing content only when a query demands it. This approach preserves query performance even when dealing with gigabytes of data.

Michael Armbrust, a key figure at Databricks, has been instrumental in driving these data innovations. The FILE type is being developed as an open standard, with efforts underway to integrate support directly into the Parquet and Delta Lake formats. This open approach ensures that the multimodal data remains portable across the entire data ecosystem, preventing vendor lock-in.

Real-World Applications

The implications for AI development are substantial. Businesses can now build enterprise document assistants that interact with contracts and policies, perform visual inspections on product images for quality control, analyze audio recordings for customer insights, or make hours of video footage searchable. For AI agents, this means having access to real evidence from diverse sources, not just text, enabling more informed decision-making and actions.

Databricks provides an example of an autonomous-driving company seeking to reduce unexplained vehicle stops. By creating a table with a FILE column pointing to dashcam videos, engineers can process these clips performantly. A Python UDF can sample frames, and an object-detection model can then analyze these frames to identify hazards. If the car stopped for no apparent hazard, that insight is surfaced, allowing machine learning teams to curate training sets for the self-driving system. All relevant data, from the video frame to extracted insights and trip metadata, resides together in a single row.

Unified Governance and Lifecycle Management

A significant challenge with unstructured data has been managing permissions and lifecycles. Storing file URLs in tables often leads to a disconnect in governance, requiring separate permission models for the table and the underlying storage. FILE type integrates with Databricks' Unity Catalog, extending row and column-level access controls to these unstructured assets. Furthermore, when a row containing a FILE is deleted, the associated binary file is also removed from object storage, simplifying compliance and data lifecycle management. This eliminates orphaned files and ensures data deletion requests are handled comprehensively.

The FILE type is designed for broad integration. It connects directly to sources like SharePoint and Google Drive via Lakeflow connectors, or it can reference data already residing in cloud storage without requiring data movement. Declarative Pipelines support incremental processing, ensuring that only new or modified documents are processed, reducing costs associated with expensive model API calls. Databricks, a company that has raised $7B and holds a post-money valuation of $134B according to StartupHub.ai data, is positioned as a major player in the data and AI infrastructure space. Its competitors, such as Palantir Technologies (StartupHub score 85/100) and Alphabet (StartupHub score 79/100), also offer advanced data management and AI capabilities, but Databricks' native FILE type offers a distinct approach to unifying multimodal data within the lakehouse architecture.

Future Developments

Databricks has outlined a roadmap that includes features for faster experimentation without duplicating data, enabling versioning and cloning of data without copying binaries. Direct streaming of datasets into PyTorch for GPU-ready tensors is also planned. Additionally, the platform aims to allow feature engineering, such as creating embeddings or classifications, without rewriting entire tables, and to support direct search and retrieval from the table itself using vector and full-text indexes. This suggests a future where unstructured data is as easily managed and analyzed as structured data, accelerating AI development across industries.