Conext Engineering for Engineers

Building effective AI systems is not about mastering “prompt engineering” or the latest RAG technique, but about the disciplined practice of Context Engineering. An AI system can be understood simply as a program: it takes an instruction set, relevant information, and user input to produce an output. The core task of a builder is to control the “relevant information” that goes into the context window to create reliable, fast, and cheap software.

The Illusion of Long Context

The industry’s push towards massive context windows—from one million to ten million tokens—is not the panacea it appears to be. While impressive on benchmarks, performance in practical applications degrades sharply long before these limits are reached.

A technical report from Chroma demonstrates this phenomenon. On simple tasks that a human could perform easily, model performance drops precipitously as the input length increases, with significant degradation observed as early as 10,000 tokens.

The primary benchmark used to substantiate long-context capabilities is the “Needle in a Haystack” test. This test is fundamentally flawed as a measure of real-world utility for two reasons:

1. Low Attention Requirement: By definition, the model only needs to find and pay attention to a single piece of information (the “needle”), ignoring the vast majority of the context.
2. Zero Reasoning Power: The task requires simple pattern matching, not complex reasoning. For example, finding the sentence “The best writing advice… was to write every week” in response to a question about the best writing advice.

Most valuable AI tasks, such as summarization or agentic workflows, require the model to pay attention to a large portion of the context and apply significant reasoning power. Relying on long context alone for these tasks leads to a significant drop in performance. Focused, curated context provides massive gains in performance compared to feeding the model the full, unfiltered context.

A Framework for Context Engineering: Gather and Glean

The central challenge is deciding what information, out of all the information in the universe, should be in the context window for any given turn. This can be approached with a two-stage model:

1. Gather (Maximize Recall): The first stage is to collect all potentially relevant information. The goal is to maximize recall, even at the expense of including some irrelevant data. This can involve creating a query plan that probes multiple sources:

   • Structured data (e.g., SQL queries)
   • Unstructured data (e.g., from a vector database like Chroma)
   • APIs and other tools
   • Web search results
   • Chat conversation history

2. Glean (Maximize Precision): The second stage is to filter the gathered data down to a pristine set of highly relevant, non-distracting information. This is a process of maximizing precision. Common techniques include:

   • Top-K vector similarity search.
   • Reciprocal Rank Fusion (RRF) to combine results from multiple retrievers.
   • Learning to Rank (LTR) models.
   • Dedicated reranking models.
   • Using LLMs themselves—often small, fast, and cheap models run in parallel—to “brute force” the search and curation process.

Context Engineering for AI Agents

These principles are even more critical for AI agents, where the gather-and-glean loop occurs repeatedly. The agent’s conversation and action history becomes a major, and rapidly growing, component of the context window. The sheer volume of code, logs, and observations generated in a few turns can be impossible for any human to parse, let alone an AI.

An interesting finding in agent performance is the value of failure:

• Giving an agent access to its past failure cases helps it break out of local minima and improve performance.
• Conversely, giving it access to prior success cases can be detrimental, causing the agent to get “lazy” and pattern-match instead of performing the required reasoning.

The Critical Role of Compaction

For agents dealing with enormous histories, compaction—distilling the essential information from the context for the next turn—is a crucial point of leverage. However, naive approaches often fail. Tests show that the compaction from some tools is negligible and can be thrown away entirely without impacting performance. A “smarter compaction with a better prompt” is required to effectively distill the context and meaningfully improve agent performance. These challenges are not yet solved, highlighting the importance of developing a shared community and practice around the discipline of context engineering.

• Tags:
• AI Agents
• LLM
• Large Language Models
• RAG
• Context Engineering
• Feature
• Retrieval
• Chroma