Why Most AI Agents Fail Horribly

Maarten Grootendorst, a Developer Relations Engineer at Google DeepMind and co-author of prominent O’Reilly books on LLMs and AI Agents, discusses the foundational concepts necessary for developers navigating today’s AI landscape. He emphasizes the need to understand the “under the hood” mechanisms rather than solely relying on prompting, especially given the democratization of AI to a broader, less technically specialized audience.

The Enduring Relevance of Foundational NLP and LLM Fundamentals

Grootendorst revisits his BERTopic library, a transformer-based topic modeling tool, noting its continued growth despite the advent of large language models (LLMs). He explains that while LLMs can perform topic modeling, BERTopic offers a more structured, standardized, and reproducible output, particularly for large datasets, which is often difficult to achieve through prompting alone. This highlights a broader trend where “NLP” and “text mining” are being supplanted by the generic term “AI,” leading many new entrants to overlook valuable “older techniques” that underpin much of the current innovation. He stresses the importance of educating developers on core concepts like embeddings and clustering, arguing that a deeper understanding of how LLMs work is crucial for developing intuition and knowing when and how to apply these powerful tools effectively.

For developers new to the field, Grootendorst identifies three fundamental concepts for LLMs:

• Tokens: Often “super underappreciated,” tokens are how LLMs perceive and interact with the world, directly impacting performance and cost (dubbed the “chief token officer” by Ben Lorica).
• Embeddings: A cornerstone for many AI applications, including BERTopic and Retrieval-Augmented Generation (RAG). Embeddings convert text into numerical representations, opening numerous possibilities for search, contextualization, and other tasks.
• Attention Mechanism: This is the core innovation enabling LLMs to contextualize information effectively, differentiating between homonyms (e.g., “bank” of a river vs. financial bank) by looking at the entire input rather than isolated words.

Deep Dive into Embeddings and Attention’s Evolution

Grootendorst elaborates on embeddings, highlighting their diverse applications. He distinguishes between:

• Contextualized embeddings: Larger, more compute-intensive, and prone to higher latency, suitable for complex contextual understanding.
• Faster static embeddings: Smaller and quicker, ideal for rapid searches, such as those performed by coding agents trawling through repositories. The choice between these depends heavily on the specific application’s needs. While leaderboards offer a starting point, nuanced factors like multilinguality, variable-size Matryoshka embeddings, and instruction-tuned embeddings further influence selection.

Discussing the attention mechanism, Grootendorst reiterates its primary function: contextualization. He then introduces State Space Models (SSMs) like Mamba as a “completely different way of approaching this attention mechanism,” offering significantly faster processing. While technically complex, SSMs are being adopted in hybrid models that interleave attention blocks with Mamba blocks to boost inference speed, which is a major bottleneck for LLMs.

The Reality and Hype of AI Agents

The definition of an “agentic” system is still evolving, but Grootendorst offers a practical view: an agent is essentially “an LLM in a for loop with some tools, some memory, and perhaps some guardrails.” This concept aligns closely with “harness engineering.” He cautions against the “hype” surrounding agents, noting that unconstrained LLMs tend to “fail horribly.” Agents require direction and constraints.

He identifies coding agents as a prime example of where agents are practically effective, significantly aiding developer productivity. However, he acknowledges that other use cases, like booking complex multi-country vacations, remain far from practical reality.

For tasks like deep research, agents can be genuinely agentic if they autonomously decide when and where to use tools (e.g., searching academic databases, summarizing abstracts) rather than following a fixed pipeline. Memory is also a critical, “underappreciated” component of agents, essential for retaining context across multi-step tasks (e.g., translating a large codebase), where the agent’s “trajectory” serves as its memory.

In enterprise settings, internal coding agents are common, particularly in privacy-sensitive sectors like healthcare and finance, where organizations prioritize control through local LLMs and custom harnesses. DevOps, data engineering, and security teams are also building agents, often with human-in-the-loop supervision. Grootendorst touches upon the risk of “vibe-coded” applications, built by non-coders relying solely on agents, which eventually break down in ways that require genuine coding expertise to fix.

The Future of Open-Weight Models, Personal Agents, and AI Infrastructure

The discussion shifts to open-weight models, which are gaining significant interest, particularly in Europe, due to a desire for ownership and control over AI technology. The performance gap between open-weight and proprietary models is narrowing, with capable models like Llama and Gemma runnable on local hardware. While the long-term supply of open-weight models faces uncertainty due to potential shifts in business decisions (e.g., Meta, Alibaba), Grootendorst believes a steady supply will persist due to the immense value they provide to the community, fostering research and new use cases. He highlights Google’s Gemma as an example of a valuable open-weight contribution.

In a rapid-fire round, Grootendorst shares his thoughts on:

• Personal Agents: Expresses enthusiasm tempered by concerns about early-stage security vulnerabilities (e.g., exposed keys, accidental deletions).
• Reinforcement Learning (RL): Calls it a “hot thing” and foundational to the current capabilities of LLMs, with ongoing advancements in efficiency and multi-turn trajectories.
• Scaling Laws: Notes their rapid evolution—from parameter count to reasoning length to trajectory length—suggesting the “nifty thing” changes quickly.

Finally, regarding data centers, Grootendorst acknowledges the global tension between the increasing demand for AI compute and local opposition to data center buildouts (due to energy, water, noise, and limited job creation). He stresses the urgent need for greater efficiency in AI, advocating for smaller, capable models like Gemma that can run on edge devices, thereby reducing reliance on massive data centers, though acknowledging that large-scale training will always necessitate significant infrastructure.

• Tags:
• AI Agents
• Large Language Models
• Embeddings
• RAG
• LLMs
• Reinforcement Learning
• Attention Mechanism
• Feature
• Scaling Laws
• AI Infrastructure
• Data Centers
• Coding Agents
• Developer Relations
• Local AI
• State Space Models
• Mamba
• BERTopic
• Topic Modeling
• Open-Weight Models