Building AI Agents That Survive Production

Haytham Abuelfutuh, CTO of Union.ai, argues that the key to production-ready AI agents is not preventing failure, but embracing it. He introduces the '3 D's' framework—Dynamic, Durable, and Defended—for building agents that can fail cheaply and recover automatically, grounded in a real-world case study of an agent system indexing over 250,000 products on Flyte.

May 14, 2026 4 min read

Building AI Agents That Survive Production — source: https://www.youtube.com/watch?v=swO5svhBhQ4

Haytham Abuelfutuh, co-author of the Flyte orchestrator, posits that the primary challenge in deploying AI agents is not perfecting their logic, but building systems that can withstand the inevitable failures of a production environment. Drawing an analogy to a human travel agent who handles dropped calls and remembers context over weeks, he notes that few developers who build agents ever ship them to production. This is because production introduces a new class of problems beyond semantic and logical errors, including infrastructure failures, network issues, API throttling, and context corruption.

The solution is not to design a perfect, infallible agent, but to build agents that tolerate failure and recover cheaply. This requires a platform built on three core design principles: the “3 D’s”.

The 3 D’s Framework for Production Agents

Dynamic: The platform must allow developers to write agents in native Python, using natural constructs like loops and try/except blocks, without being constrained by a restrictive Domain-Specific Language (DSL). This enables an intuitive and flexible development experience.
Durable: Agents must be resilient to crashes, out-of-memory (OOM) errors, and infrastructure events like spot machine preemption. The key is to make failures cheap and recovery seamless.
Defended: The system must provide a secure way to handle untrusted, agent-generated code and include mechanisms for human-in-the-loop intervention when the agent is stuck.

Deep Dive: Durability and Crash Recovery

Durability is critical for long-running agent sessions that can last for weeks. The platform must support robust crash recovery by being “infrastructure-aware.”

Infrastructure as Code: Developers should declare infrastructure requirements (e.g., 32 GB of RAM, specific CPU counts) directly within their Python code. When an OOM error occurs, the agent is aware of its environment and can be retried on a machine with more memory. The code itself controls both the logic and the compute resources.
Checkpointing and Caching: To recover from a crash, the system must record every action the agent takes. This is especially important for non-deterministic calls.
- LLM Calls: Calling an LLM with the same prompt may yield different results over time. Caching the original response ensures that upon recovery, the agent behaves deterministically from the last checkpoint.
- User Interactions: If an agent asks a user for their budget and then crashes, it must not ask the same question again upon recovery. Caching user inputs is essential for a good user experience.
Cross-Session Caching: The platform should also allow for sharing cached results across different users and sessions. For example, a common web search query can be cached and reused, saving tokens and improving latency for subsequent runs.

Deep Dive: Defended Execution

Agents will inevitably encounter situations where their pre-defined tools are insufficient. In these cases, they may need to generate and execute their own code.

Sandboxed Environments: This agent-generated code is untrusted and must be run in a secure, sandboxed environment that is network- and host-isolated. The talk gives a shout-out to Pydantic’s monty project for providing this capability.
Iterative Code Correction: If the generated code fails, the platform should capture the error. The agent can then feed the original code and the error back to the LLM, asking it to fix the bug. This iterative loop can be retried several times until the code executes successfully.
Human-in-the-Loop: If the agent fails to generate working code after a set number of retries, it should be able to “bail out” and prompt a human user for help, presenting options or asking for clarification.

Case Study: Dragonfly

Dragonfly built a product recommendation agent on Union.ai’s platform to help users choose software from a catalog of over 250,000 products. Their four-tier agent architecture (catalog, coordinator, researcher, tools) was able to move from a laptop prototype to a full-scale production system in a single session. The coordinator agent is even capable of identifying redundant researcher agents and re-tasking them, demonstrating a sophisticated, durable, and scalable design.

Q&A Highlights

Technology Stack: The Union.ai platform backend is built with Go and Rust for performance, while developers interact with it via a Python SDK that includes Rust components.
Shifting Infrastructure Left: For platform teams hesitant to let developers define infrastructure in code, the key is to demonstrate the massive increase in iteration velocity. When developers can control their own environments, the cycle of building, packaging (e.g., Helm charts), and deploying through a central team is eliminated, allowing them to experiment and ship much faster.