Context Engineering & Agentic Search with the CEO of Chroma
Jeff Huber, CEO of Chroma, discusses "context rot," the degradation of AI performance in large context windows, and outlines a new vision for retrieval infrastructure. He covers the evolution of search, the importance of a two-stage recall-then-precision pipeline, and the challenges of agentic memory, advocating for a shift from AI "alchemy" to reliable engineering.
Context Rot and the Limits of Large Context Windows
The excitement around massive, million-token context windows often overlooks a critical problem: context rot. This refers to the degradation of a model’s performance as the context length increases. Research from Chroma validated the intuition many developers already had: a model’s ability to pay attention and reason declines with more context.
While benchmarks like “needle in a haystack” show high performance, they are often simple lexical matching tasks that don’t require reasoning. Real-world applications, however, demand connecting multiple pieces of information and reasoning about them, which is where models begin to fail.
This performance degradation highlights the “spiky” nature of AI intelligence. Models are not uniformly capable across all tasks and conditions. The intense competition between AI labs creates a negative incentive for transparency about these weaknesses. Instead of acknowledging that models have specific strengths and weaknesses, the narrative often pushes towards a belief in an emergent, generalized intelligence, which doesn’t align with the current reality. The intellectually honest approach is to map out these spikes—what models are good at and what they are not—to build reliable systems.
The Evolution of Search: From Vector Databases to Modern Retrieval Infrastructure
The term “vector database” is a limiting, VC-driven meme. The actual problem developers need to solve is broader: information retrieval. Dense vector search is just one tool in the toolbox, not a panacea. Chroma positions itself as “modern retrieval infrastructure for AI,” built to address the specific needs of AI-driven applications.
This “modern” approach differs from legacy search systems in several key ways:
- Workload Shape: Classic search involves one giant index queried by many users. AI workloads consist of many smaller, distinct indexes—per user, per team, per workspace.
- The Querier: Humans, who can only process about “10 blue links,” are being replaced by LLMs. An LLM can issue hundreds of queries and process thousands of documents for a single task, fundamentally changing the scale and nature of retrieval.
- Developer Accessibility: Search is no longer the exclusive domain of specialized search engineers. Every developer, including semi-technical users, is now building with retrieval systems.
A Two-Stage Retrieval Pipeline: Maximizing Recall, then Precision
To build effective retrieval systems that serve LLMs, a two-stage pipeline is essential.
Stage 1: Maximize Recall
The first goal is to gather all potentially relevant information. Missing a single critical piece can lead to an incorrect outcome. This stage requires a thoughtful combination of different search techniques, as no single method is universally superior.
- Lexical (Full-Text) Search: Excels when you know the specific terms to search for, common in enterprise search where a user remembers keywords from a document.
- Dense (Semantic) Search: Powerful when you don’t know the exact language of the corpus but need to find information based on an idea or concept.
Most applications benefit from a combination of these, tailored to the specific use case. The recent discourse pitting BM25 against dense embeddings often lacks nuance; the effectiveness of each depends entirely on the nature of the queries.
Stage 2: Maximize Precision
Once a wide pool of information is gathered, the next step is to filter it down to only the most relevant, high-signal pieces before passing it to the model. This is critical because of context rot and the model’s sensitivity to distractors.
“The most insidious distracting information is the information that looks relevant but is not for some subtle reason.”
Feeding a model more information can often confuse it, especially if that information is subtly incorrect or irrelevant, leading to worse outcomes.
Challenges in Agentic Search and Memory
These retrieval principles are paramount for building AI agents. A common failure mode for agents is getting stuck in a local minimum. When given examples of prior successful tasks, an agent might blindly copy the solution without reasoning, especially if the new task is subtly different from the example.
This touches on the unsolved problem of agent memory, which can be framed as a “context engineering” challenge. It involves:
- A Write Path: Gleaning and storing insights from prior contexts.
- A Read Path: Effectively using those prior insights in new situations.
The goal is to enable models to build a “library of skills” from tacit knowledge. While storing skills as documents in a search index is part of the solution, a learned representation encoded into the model’s weights will likely be necessary.
The concept of sub-agents is a powerful architectural pattern to combat context rot, as it enforces “context window encapsulation.” Each sub-agent operates within its own limited context, performs a specific task, and then compacts its findings before reporting up to an orchestrator agent.
Context Hygiene and On-Device Retrieval
In any multi-turn conversation or agentic loop, context cleanliness is crucial. Naively summarizing the entire history has been shown to be no better than simply clearing the context and starting fresh.
More effective strategies for managing context include:
- Prompted Compaction: Using a detailed prompt to help the model identify and extract only the most relevant pieces of information for the next turn.
- Historical Search: Giving the agent the ability to search over its own history for specific, relevant information.
Recognizing the need for privacy, speed, and offline capability, Chroma has been released for on-device use (Swift and Android). Because it’s written in Rust, Chroma can run anywhere—from a Jupyter notebook to a distributed cloud environment to mobile devices and even robots—all with a consistent API. This allows developers to build applications where retrieval happens locally, ensuring user data never leaves the device unless explicitly desired.
Vision: From AI Alchemy to Reliable Engineering
The ultimate goal is to move AI development from its current state, which often feels like alchemy, to a more robust and predictable engineering discipline. The famous XKCD comic of a machine learning system being a “pile of garbage” that you “just stir” when it doesn’t work still rings true.
AI is a powerful tool, but it’s only useful if given the right context and the ability to learn. The current frontier involves creating systems that can learn from experience to handle the “fractally complex” nature of the real world. Without this, developers will continue getting “mugged by reality,” endlessly patching edge cases just as they did in earlier eras of computer vision and autonomy. The future lies in building reliable, adaptable systems that can overcome the “high-interest credit card of technical debt” that machine learning has historically represented.