Databricks Lakehouse: The AI Context Layer

Cellcentric, a joint venture between Daimler Truck and Volvo Group, is tackling a core challenge in industrial AI: integrating scattered research and development data into a trustworthy AI context. Their solution, the Data Hub, built on Databricks, acts as a governed AI context layer, unifying data from sources like IoT telemetry, SAP, and manufacturing execution systems (MES).

This approach is crucial because AI agents, much like human engineers, need to reason over data with a complete understanding of its origin, meaning, and limitations. As detailed on the Databricks blog, the Data Hub leverages Unity Catalog for governance and Lakehouse Federation to incorporate on-premises data, creating a robust foundation.

The Fuel Cell Passport: A Data Product Example

A key data product within the Data Hub is the Fuel Cell Passport. It integrates data from five enterprise systems, modeling seven hierarchy levels and supporting detailed historical analysis. Daily quality checks ensure its readiness for engineering, quality, and manufacturing investigations.

This internal engineering data product exemplifies the traceability and lifecycle concepts of a product passport, though distinct from regulatory artifacts.

Context as a Quality Metric

Beyond traditional data quality metrics like completeness and freshness, the Data Hub emphasizes context coverage. This includes detailed markdown catalog entries explaining the data product's purpose, usage, and caveats.

This focus has shifted documentation from an afterthought to a first-class quality metric, with published data products averaging 90% column-comment coverage, often AI-assisted and human-reviewed.

Unified Access for Humans and Agents

The Data Hub provides a single interface: a marketplace and workbench for employees, and an MCP (Model Communication Protocol) server for AI clients. Both human users and AI agents access the same governed data and context.

Identity management is centralized through Azure AD, flowing into Databricks via OAuth 2.0, ensuring agents operate within the user's permission boundaries. This prevents agents from creating a secondary, less secure access path.

Governance and Observability

The architecture is built on a unified operating model where identity, data access, and tool usage are governed and observable. Unity Catalog enforces authorization, while Unity AI Gateway and MLflow tracing provide visibility into model and tool calls.

A continuous evaluation framework assesses agent performance, tool interactions, and the accuracy of the context layer itself, ensuring the AI's reasoning remains aligned with domain expertise.

This comprehensive approach dramatically accelerates R&D investigations, reducing them from weeks to days.