Multi-Agent Orchestration Patterns

Sandipan Bhaumik, Data & AI Tech Lead at Databricks, shared insights into the critical patterns for managing multi-agent AI systems in production. With over 18 years of experience in building and scaling distributed data systems, Bhaumik highlighted common mistakes and offered best practices for orchestrating AI agents, particularly in regulated industries like financial services and healthcare.

The Problem with Scaling AI Agents

Bhaumik began by illustrating the exponential increase in complexity when moving from a single-agent system to a multi-agent one. He noted that while a single agent is often perceived as a feature, a system with multiple agents transforms into a distributed systems problem. This is due to the inherent challenges of coordination, state management, and failure recovery that arise with increased agent interactions.

He presented a "complexity curve" showing how coordination complexity grows disproportionately with the number of agents. For instance, five agents can be 25 times more complex to manage than a single agent. This complexity can lead to numerous failure modes, including race conditions and state synchronization issues.

A "War Story": The Race Condition

To illustrate these challenges, Bhaumik shared a "war story" about a credit scoring system that failed due to a race condition. In this scenario, a credit score calculation agent successfully wrote a score of 750 to the database. However, a subsequent risk assessment agent, operating with stale data from a cache, read an outdated score of 680, leading to incorrect decisions.

The root cause was identified as a caching layer without proper invalidation. This highlights a common anti-pattern: building a distributed system without embracing distributed systems thinking. The problem wasn't the LLM or the prompt, but the underlying system architecture.

Choreography vs. Orchestration: Choosing the Right Coordination Pattern

Bhaumik then delved into two primary patterns for coordinating multi-agent systems: choreography and orchestration.

Choreography: Event-Driven Coordination

In choreography, agents coordinate through events and a message bus. Each agent is autonomous and loosely coupled. For example, a Research Agent publishes a "research_completed" event, which an Analysis Agent subscribes to. The Analysis Agent then performs its task and publishes an "analysis_ready" event, which a Report Agent consumes. This pattern offers high autonomy and is suitable for workflows where agents are frequently added or modified. However, debugging can be difficult due to the lack of a centralized control flow.

Orchestration: Centralized Coordination

Orchestration involves a central orchestrator that directs the workflow by calling agents sequentially or in parallel. Agents in this model are typically "dumb," executing specific tasks based on the orchestrator's commands. This approach provides better visibility, control, and easier debugging, making it suitable for complex, stable, and enterprise-grade workflows.

When to Use Which Pattern

Bhaumik presented a decision matrix based on workflow complexity and autonomy requirements:

• Choreography is ideal for loosely coupled workflows, high agent autonomy, and situations where agents are frequently added. It requires strong observability infrastructure, and debugging can be challenging.
• Orchestration is preferred for complex dependencies, when centralized rollback capability is needed, and for stable workflows. It can suffer from bottlenecks if not managed properly and may offer weaker autonomy.
• Hybrid patterns can be used for distributed transactions that require a blend of both approaches.

He stressed that a key consideration is the ability to trace events and understand the flow of data, especially when multiple agents are involved.

State Management: Immutable Snapshots

A critical aspect of robust multi-agent systems is state management. Bhaumik advocated for using immutable state snapshots. Each agent works with a versioned, immutable state object. When an agent completes its task, it produces a new state version and passes it to the next agent. This approach:

• Validates the contract between agents.
• Ensures immutability, preventing accidental modifications.
• Provides a clear lineage for audit and replay, aiding debugging.

He cautioned against the anti-pattern of shared mutable state, where multiple agents directly modify a common data store, leading to race conditions and unpredictable behavior.

Failure and Recovery Patterns

Acknowledging that failures are inevitable, Bhaumik highlighted two key patterns for handling them:

1. Circuit Breaker Pattern: This pattern prevents cascading failures by monitoring agent calls. If an agent consistently fails (e.g., 5 times), the circuit breaker "opens," preventing further calls to that agent for a timeout period. This protects the system from overload and allows the failing agent to recover. A "half-open" state is introduced to test availability periodically.
2. Compensation Pattern (Saga Pattern): For workflows involving multiple steps, this pattern ensures atomicity. If a step in the workflow fails, subsequent steps are undone by running compensating actions in reverse order. For example, if an "Execution Agent" fails, the "Analysis Agent" might delete its draft recommendation, and the "Research Agent" might clear its cached data. This ensures that the system remains in a consistent state even after failures.

Production Architecture Considerations

Bhaumik illustrated a typical production architecture for multi-agent systems, emphasizing the role of an orchestrator like LangGraph or a similar workflow engine. This orchestrator manages the workflow graph, state store, and observability, ensuring that agents execute correctly and that failures are handled gracefully. Data is stored in Delta Lake for its ACID properties and schema enforcement capabilities.

He concluded with three key takeaways:

1. Agent chaos is inevitable when scaling; focus on building robust systems, not just demos.
2. Choreography is a choice, but orchestration offers more control and simplifies debugging.
3. Immutable state snapshots and circuit breakers are crucial for building reliable multi-agent systems.