Databricks: The AI Playbook for Enterprise Agents

Sandipan Bhaumik, Data & AI Tech Lead at Databricks, recently shared insights on "The Production AI Playbook: Deploying Agents at Enterprise Scale." In his presentation, Bhaumik outlined a critical framework for organizations looking to move beyond experimental AI to production-ready agents. He emphasized that simply selecting the right model is not enough; a comprehensive approach is necessary to ensure successful deployment and ongoing management.

The Problem Pattern in AI Deployment

Bhaumik highlighted a common, often frustrating, pattern he observed across numerous customer conversations. This pattern typically involves a rapid development cycle where models are picked, features are built to look good, and demos are presented to leadership, leading to a quick sign-off. However, this often culminates in a critical question like "Why is AI botching us?" and the realization that a significant percentage of AI projects fail in production.

The core issue, according to Bhaumik, stems from a lack of focus on fundamental aspects like observability, evaluation, and governance. He pointed out three key gaps that prevent AI systems from being production-ready:

• Observability Gap: "You can't debug what you can't see." Without robust observability, it's impossible to understand why an AI system is failing or behaving unexpectedly.
• Evaluation Gap: "You can't improve what you can't measure." Without clear, quantifiable metrics, it's difficult to gauge performance and identify areas for improvement.
• Governance Gap: "You can't trust what you can't explain." Lack of clear accountability and governance makes it challenging to build trust in AI systems, especially in regulated industries.

Bhaumik stressed that these are not merely nice-to-haves but essential components for any AI system intended for production. He illustrated this with a case study of a retail banking chatbot project that aimed to handle customer queries about account balances and overdraft fees. The initial approach focused heavily on model selection and feature development, leading to a successful-looking demo.

The Five Pillars of Production AI

To address these challenges, Bhaumik presented a five-pillar framework for building production-ready AI agents:

1. Evaluation First: This involves defining success metrics with numbers before any code is written. It requires building test cases from real call logs and automating the grading process. The goal is to measure success before you have anything to measure.
2. Observability: Bhaumik stated, "If you can't replay a failed conversation in under 5 minutes, you're not production ready." This pillar emphasizes the need to see everything, always, by collecting detailed traces of agent interactions.
3. Data Foundation: He noted that 60% of project time is spent here, yet many teams skip it. This involves building robust question data (what the AI needs to answer) and tracking data (what the AI does). Real-time APIs, versioned knowledge bases, and customer history are crucial for this.
4. Orchestration: This pillar focuses on how to manage and coordinate multiple AI agents. Bhaumik outlined three patterns: Orchestrator-Worker for complex workflows, Choreography for independent tasks, and Human-in-the-Loop for regulated environments with high-stakes decisions.
5. Governance: Bhaumik defined governance not as what stops shipping, but as what pulls you back when things go wrong. This includes audit trails, PII pre-validation, prompt versioning as change management, and model change management.

Building for Success: A Case Study

Bhaumik then detailed how his team applied these principles to the retail banking chatbot case. Instead of picking a model early, they focused on the evaluation layer first, defining success metrics and building test cases. They then established a data foundation by collecting and structuring relevant data. Only after these foundational steps did they select models and build the AI layer, focusing on orchestration and governance.

The results, six weeks post-launch, were significant: 87% accuracy, a 62% deflection rate, 65% response time improvement, and a CSAT score of 4.4/5. Crucially, the observability and data foundation allowed them to quickly identify and fix a policy update bug that was propagating incorrect information, saving significant time and effort.

Bhaumik concluded by reiterating the importance of measurement and iteration, emphasizing that "Observable, Debuggable, Improveable. That's what you get when you build measurement first." This structured approach, he argued, is key to successfully deploying AI agents at enterprise scale.