Why AI Agents Fail
Abi Aryan discusses the shift from MLOps to LLMOps, framing generative AI deployment as a distributed systems challenge. She highlights the critical role of FinOps due to high model costs, the need for custom tooling over generic agent frameworks, and the importance of a 'decision layer' for agent autonomy. Observability is re-conceptualized as anomaly detection, and she points to dynamic RL environments and strong data engineering as future research priorities and essential skills for AI engineers.
The discussion with Abi Aryan, author of “LLMOps: Managing Large Language Models in Production,” centers on the evolution of MLOps into LLMOps, emphasizing that deploying generative AI has largely become a distributed systems problem.
The Evolution from MLOps to LLMOps
Abi highlights a significant shift in infrastructure and pipelines compared to earlier ML systems. While ML Ops focused on containerizing models for production, LLMOps is fundamentally different, primarily driven by the scale and non-deterministic behavior of large language models. The role of an LLMOps engineer now heavily involves FinOps, controlling the massive costs associated with these models, and managing performance in terms of latency, throughput, and memory across diverse tasks. The focus has moved from merely building systems to building systems that perform not just technically, but also behaviorally. Traditional software engineers, familiar with concepts like strong cohesion, low coupling, and microservices, find the transition to LLMOps easier than traditional MLOps engineers, as LLMs are viewed akin to “software 3.0”—an end-to-end system where the model is the product.
Agents and Agentic Workflows as Distributed Systems
Agents are defined as components or models that aim to understand user intent, even if implicitly stated. Agentic workflows, conversely, are responsible for planning and orchestrating what these agents should do. This architecture means teams are moving away from monolithic models towards distributed systems, where agents require dynamic provisioning, effective communication, and robust testing across different tasks. This paradigm introduces challenges such as “compounding reliability,” where the failure probability increases across multiple interconnected agents in a workflow. Abi notes that most companies initially adopted monolithic architectures but are transitioning to microservices to scale. A key challenge is agent consensus, especially when agents are distributed across varying hardware (e.g., H100 vs. other GPUs), making it difficult to achieve consistent outcomes if one node fails.
Limitations of Agent Frameworks and Human Workflow Replication
Many vertical AI companies initially experimented with agent frameworks (e.g., CrewAI, LangChain) but eventually outgrew them, opting to build custom tooling due to the unreliability and bloat of generic frameworks. Similarly, general-purpose evaluation tools were found to be insufficient, leading companies to develop in-house evaluation frameworks. Abi criticizes the common approach of trying to replicate human workflows directly into AI agents, calling it an attempt to “replicate intelligence without understanding what intelligence is.” She argues that human processes, often involving inefficient communication (e.g., five emails), are not always the most efficient and that agents are not merely “glorified Zapier if-then routines.”
Agent Autonomy and the Missing Decision Layer
Agents operate on different levels of autonomy; most enterprises begin with significant human involvement, gradually increasing autonomy as confidence grows. A critical missing component in current LLM-based agents is a decision layer. LLMs excel at knowledge retrieval but lack independent decision-making capabilities. Abi emphasizes the need to develop symbolic language and models of the world for agents, allowing them to understand cause and effect, prioritize decisions, and articulate their reasoning. This decision-making reasoning layer is crucial for agents to move beyond being mere knowledge retrieval tools.
Observability and FinOps as Anomaly Detection
Observability in LLMOps should transcend simple visibility and be approached as an anomaly detection problem. While existing tools can be utilized, more extensive logging and custom logs are necessary. Abi advocates for building simple ML models to detect anomalies across various metrics, treating it like a “fraud detection pipeline” for agent behavior. FinOps, or managing AI costs, is still in its nascent stages, with less than 5% of companies effectively addressing it. A significant challenge is accurately profiling models to understand cost origins—whether they are compute-bound (RAM-intensive, solvable with kernel optimization) or memory-bound (solvable with data engineering pipelines). There’s also the communication-bound problem, optimizing GPU communication.
The Economics of Inference and Reinforcement Learning
Effective FinOps requires mastering token economics. While KV caching is common, many struggle to differentiate what should be stored in cache versus RAM, impacting throughput. In reinforcement learning (RL), a major limitation is the use of static environments. Agents learn dynamically, and current RL environments often fail to adapt in real-time, leading to rapid obsolescence of learned models. Abi suggests the potential of curriculum learning to dynamically adjust task difficulty in RL environments.
Future Research Areas and Essential Skills
Three key research areas to watch in the next 6-18 months include:
- Profiling ML models and entire systems: Understanding the system as a whole, not just isolated models.
- Curriculum learning applied to reinforcement learning: Enabling RL environments to adapt dynamically.
- Data engineering: A critical foundational area where significant gains can still be made.
For AI/ML/LLMOps engineers, the essential skills for the next 18-24 months are:
- Learning to profile models effectively.
- Understanding distributed systems, with tools like Ray being a good starting point (or vLLM). The focus should be on grasping distributed systems concepts before diving deep into models.