From Chaos to Choreography: Multi-Agent Orchestration Patterns That Actually Work — Sandipan Bhaumik
Sandipan Bhaumik from Databricks explains that scaling from one to many AI agents is a distributed systems problem, not an AI one. He details common architectural anti-patterns like shared mutable state that cause race conditions and silent failures. The talk provides a practical framework based on distributed systems engineering, covering crucial patterns like choreography vs. orchestration, immutable state management with versioning, data contracts, and failure recovery using circuit breakers and compensation (Saga) patterns. Bhaumik illustrates how to build a robust, production-grade multi-agent architecture using tools like Databricks, LangGraph, and MLflow.
The Core Problem: Multi-Agent Systems are Distributed Systems
Scaling from one to many AI agents is fundamentally a distributed systems problem, not an AI model or prompt engineering problem. Teams often treat adding agents as merely adding features, which leads to coordination chaos, silent failures, stale data, and untraceable decisions. The complexity grows exponentially; a system with five agents has at least ten potential connections, each representing a point of failure, a potential race condition, or a state synchronization issue.
A production example from a financial services credit decisioning system illustrates this peril. A five-agent system began making incorrect approvals. The root cause was a race condition involving a caching layer: a credit score agent updated the database, but cache invalidation failed. A subsequent risk assessment agent read the stale score from the cache, leading to a flawed decision. The lesson learned was that project failure often stems from “bad architecture,” not “bad AI.”
Architectural Patterns for Coordination
The first critical decision is choosing a coordination pattern. Two fundamental approaches exist:
-
Choreography (Event-Driven): Agents are decentralized and autonomous, communicating via events on a message bus. One agent publishes an event (e.g.,
research_completed), and another subscribes to it. This pattern offers loose coupling, high autonomy, and scalability. However, it is a “nightmare to debug” without bulletproof observability to trace event flows. It is best suited for simple, naturally event-driven workflows where high agent autonomy is a priority. -
Orchestration (Centralized): A central orchestrator (e.g., using LangGraph) manages the entire workflow as a Directed Acyclic Graph (DAG). It calls agents directly, manages state, handles parallelism, and executes retries. Agents are kept simple and do not communicate with each other. This provides a single source of truth, making it easy to debug, trace, and implement rollbacks. It is ideal for complex workflows with low autonomy tolerance, such as in financial services.
A decision framework based on workflow complexity and autonomy requirements can guide the choice, with hybrid patterns like Choreography with Sagas being an option for complex, high-autonomy scenarios.
State Management and Data Contracts
Proper state management is crucial for avoiding chaos as you scale.
-
Anti-Pattern: Shared Mutable State: Allowing multiple agents to write to the same database records concurrently leads to lost updates and race conditions. Default database isolation levels are often insufficient without explicit locking.
-
Solution: Immutable State Snapshots with Versioning: The recommended pattern is for agents to produce immutable, versioned state objects. State is stored as an append-only log, meaning new state versions are inserted, not updated. This eliminates race conditions and provides a clear, auditable lineage of state evolution, which is invaluable for debugging and rollbacks.
-
Data Contracts: To ensure clean handoffs, agents should adhere to formal data contracts that define their input and output schemas. This allows for validation at each step, catching data quality or schema mismatch issues immediately. On Databricks, these contracts can be registered and governed in Unity Catalog.
Failure Recovery Patterns
Agents will inevitably fail. A robust system must be designed for failure.
-
Circuit Breaker Pattern: This pattern wraps calls between agents. If a call fails repeatedly, the circuit “opens,” causing subsequent calls to fail fast instead of timing out. This prevents a single failing agent from causing cascading failures across the system. After a cooldown period, the circuit enters a “half-open” state to test for recovery.
-
Compensation (Saga) Pattern: Each agent implements both an
executeand acompensatemethod (to undo its action). If an agent fails mid-workflow, the orchestrator calls thecompensatemethod for all previously completed agents in reverse order. This provides a robust rollback mechanism, ensuring the system returns to a clean state with no partial transactions left behind.
A Production-Grade Architecture on Databricks
A reference architecture using the Databricks platform can implement these patterns:
- Orchestration: LangGraph integrated with the Mosaic AI Agent Framework.
- Agents: Implemented as versioned and governed Unity Catalog functions or models.
- Serving: Databricks Model/Function Serving with AI Gateway to enforce policies like circuit breakers.
- State Management: Immutable state versions stored as an append-only log in a Delta Lake table.
- Observability & Governance: MLflow for tracing calls and Unity Catalog for access control and lineage.
Final Thoughts
When building multi-agent systems, remember that agent chaos is inevitable without proper planning. The key is to make a conscious architectural choice, adopting proven distributed systems patterns. The goal is to build robust, reliable systems, not just impressive demos. The “unsexy” infrastructure work—implementing circuit breakers, immutable state, and compensation logic—is what creates dependable, production-ready applications that deliver sustained business value.