Databricks Adds OpenTelemetry Tracing

Databricks is enhancing its observability capabilities by enabling direct ingestion of OpenTelemetry (OTel) traces into its Unity Catalog. This move aims to address the challenges of managing the vast amounts of trace data generated by AI agents, a problem that traditional observability tools struggle to handle cost-effectively and with adequate governance.

As AI applications mature and move into production, understanding agent behavior through trace data, capturing prompts, tool calls, responses, and latency, becomes critical. Without robust tracing, debugging, evaluation, and governance become significantly more complex. The ability to retain and analyze these traces longer, join them with other business data, and reuse them for evaluation is paramount.

AI Tracing Challenges Traditional Observability

Traditional observability platforms often present limitations when dealing with the scale and sensitivity of AI agent traces. High retention costs, fragmented governance, and the necessity for extra data pipelines to integrate traces into analytics workflows are common pain points. Sensitive prompt data further complicates sending traces to third-party SaaS tools, creating InfoSec friction and data sovereignty concerns.

Databricks' new integration shifts trace data into the Lakehouse, treating it as a first-class dataset. This allows teams to query, dashboard, and build ETL pipelines using familiar SQL tools, while also applying granular governance controls like PII masking.

Serverless Ingestion Simplifies Telemetry

The platform introduces a fully managed, serverless ingestion path powered by Zerobus Ingest. This engine natively supports standard OpenTelemetry protocols (OTLP) via gRPC and a REST API, allowing direct export of spans, logs, and metrics from OTel-compatible collectors and application frameworks like MLflow. This eliminates the need for intermediate message buses like Kafka and reduces operational overhead.

This architecture provides a "single-sink" approach, streaming telemetry data directly to the Lakehouse. It supports high-throughput ingestion and long-term retention without the cost pressures often associated with SaaS observability solutions.

From Debugging to Deeper Analytics

By landing traces directly in Unity Catalog, Databricks enables teams to move beyond basic debugging. Production traces become immediately usable for analytics, facilitating faster iteration loops between real-world usage, model evaluation, and continuous improvement. The MLflow evaluation stack is enhanced, allowing for large-scale offline evaluations and continuous monitoring of production systems.

The integration also introduces native observability dashboards within the MLflow Experiment UI, offering insights into trace volume, errors, latency, token usage, and cost. This unified approach to observability aims to create a continuous improvement flywheel for AI agents.

Databricks Tames AI Agents with new Lakehouse observability features.