Databricks Tames Agentic AI

The era of AI agents orchestrating multi-step workflows across disparate systems is here, but it’s a governance nightmare. Databricks is stepping in with an expanded AI Gateway, aiming to bring order to the chaos of agentic AI. This updated platform provides a unified governance layer, tackling the critical need for control and auditability in increasingly complex AI deployments.

StartupHub data

Companies working on this

Profiles of the companies named in this story, with funding and a one-liner from our database.

Databricks

$190.0B

A unified data analytics and AI platform built on the lakehouse architecture.

Google

$9.1B

Global technology leader in search, advertising, cloud, AI, and consumer electronics.

Anthropic

Private / $100B+ est

Anthropic is an AI safety and research company building reliable, interpretable, and steerable AI systems, best known for the Claude family of models.

OpenAI

Private / $100B+ est

OpenAI is an AI research and deployment company dedicated to ensuring that artificial general intelligence benefits all of humanity.

Agentic AI, where models interact with tools, APIs, and other systems to complete tasks, presents significant challenges. Traditional governance tools, built for siloed applications, fall short. Databricks' approach aims to span the full lifecycle of an agent's actions, from LLM access to external system interactions.

Unified Control for Complex Workflows

The core of the update is extending AI Gateway to manage how LLMs interact with tools like APIs and coding assistants. This includes new support for governing MCP (model communication protocol) usage, allowing organizations to dictate which agents can access which external systems and monitor that data usage. This move is a significant step toward comprehensive LLM guardrails, moving beyond simple LLM access to encompass the entire agentic ecosystem.

Databricks is enabling users to set up LLM endpoints and MCP servers in seconds, supporting a range of models from Anthropic, OpenAI, Google, and open-source options. The key benefit is consistent policy application across providers, eliminating the need for duplicate configurations.

Fine-Grained Permissions and Guardrails

Preventing unwanted actions is paramount. AI Gateway introduces fine-grained access control for tools, supporting on-behalf-of user execution for MCP calls. This ensures agents operate with the same permissions as the requesting user, preventing unauthorized data access.

Flexible guardrails, powered by an LLM-judge approach, offer customizable protection. These can be applied to requests, responses, or both, covering PII detection and redaction, content safety, prompt injection defense, data exfiltration prevention, and hallucination checks. This offers robust Agentic AI control.

Each guardrail is configurable with custom prompts and models, allowing for dynamic policy enforcement. Violations can result in request rejection or data masking, with all actions logged for audit.

End-to-End Observability

Understanding AI agent behavior is crucial for FinOps, engineering, and security teams. AI Gateway provides unified logging infrastructure for all three.

FinOps teams can track costs with detailed attribution by endpoint tags, request tags, identity, model, and provider. This goes beyond token counts to include actual dollar costs for provisioned throughput and external model pricing.

Engineering teams gain access to full payloads for debugging inference tables, capturing request/response details, latency, and errors. This allows for rapid troubleshooting when agents fail.

Security teams benefit from complete audit trails, logging requesting identities, timestamps, and MCP call details. Unity Catalog permissions control data access to these logs.

Production-Ready Reliability and Flexibility

Databricks is emphasizing production readiness with features designed for flexibility and resilience.

A new OpenAI-compatible API allows for seamless provider switching without code changes. Applications can be written once and then point to different model endpoints as needed.

Automatic failover capabilities ensure continuous operation. Users can configure fallback models, allowing requests to route to backup options if primary models encounter rate limits or errors. This is critical for maintaining service level agreements (SLAs).

Rate limiting at the endpoint, user, or group level is also available to prevent runaway costs and protect system stability.

These capabilities are now available in supported Databricks regions, building on the company's broader efforts in enterprise AI platforms, such as the recently launched Databricks Agent Bricks platform.