Context Engineering for AI Agents: 4 Failure Modes to Avoid
Martin Keen explains why bigger context windows can make AI agents less reliable. He introduces context rot, the four context failure modes of poisoning, distraction, confusion, and clash, and the context engineering fixes that work by removing information rather than adding it.
Bigger Windows Do Not Automatically Make Better Agents
The video starts from a counterintuitive premise: context windows have grown from a few thousand tokens to a million or more, yet more context often makes AI agents worse rather than better. A recent study of 18 frontier models found that every one of them became less reliable as the context window grew. Researchers have named this context rot: the more tokens in the window, the less reliably the model uses any of them. Context engineering is presented as the discipline meant to counter this, defined as deciding what actually goes into the context window at each step of an agent’s run. The central argument is that the fixes for these failures involve deliberately throwing information away, so better agents often require less context, not more.
Anatomy of an Agent’s Context Window
To make this concrete, the video follows a hypothetical migration agent tasked with moving an aging codebase to a new database driver. Its context window fills up in roughly this order:
- System prompt: standing instructions for how the agent operates in general.
- Task prompt: the specific job, often only a few lines long.
- Project instructions: files such as agents.md that explain how the project is built, pulled in by many agent harnesses.
- Tool definitions: a description of every tool the agent can call.
- Message history: a running log of every step the agent has taken, which grows substantially over an hour of operation.
- Tool results: often the largest component, including file contents, test output, error logs, and schema dumps.
The key observation is that the actual job description occupies only a few lines near the front, while nearly everything else is material the agent piles up along the way. Buried in that material is a note the agent wrote for itself, claiming the new driver retries failed connections automatically, so the old retry code could be deleted. The speaker flags this note deliberately, since it becomes the example for the first failure mode.
The speaker explains why size matters through the idea of an attention budget. A model does not look things up in its context the way a database retrieves a row. Every token competes for a finite amount of attention, so a window holding a million tokens does not provide a million tokens’ worth of attention.
The Four Failure Modes
- Context poisoning: An error enters the window and gets treated as fact, because to the model its own notes look like ground truth. In the example, the agent skimmed the driver documentation, misread it, and recorded a false conclusion about automatic retries. It then deleted perfectly good retry code, file after file.
- Context distraction: A long history becomes the strongest pattern in the window. Since a language model is essentially a pattern continuation machine, an hour of the agent’s own past actions can overpower the reasoning it learned in training. Instead of asking what the job needs next, the model drifts into repeating what it has already done and can get stuck going over old ground.
- Context confusion: Every tool definition sits in the window whether it is used or not, and having more tools makes choosing the right one harder. In one benchmark, a model failed a task when offered 46 tools but passed the very same task when offered only 19. The speaker calls this tool sprawl.
- Context clash: The window contains information that contradicts itself. Eventually the agent notices that no retries are happening and records that the earlier note was wrong, but now both the false and corrected versions sit in the window, presenting conflicting information.
Fixes: Throwing Information Away
Each failure mode has a matching remedy:
- Validate against poisoning: check conclusions against reality before writing them down. A single connection test would have prevented the bad retry note.
- Compact against distraction: when the history grows too long, summarize it, discard the full log, and restart from the summary. The speaker cautions that compaction is lossy, and a summary of a summary can drift like a photocopy of a photocopy. A stronger approach is to move as much as possible out of the window into permanent files, pulling detailed information in only when it is actually needed.
- Curate against confusion: give the agent only the handful of tools the job requires, while allowing it to fetch others when necessary. A database migration does not need 46 tools.
- Prune against clash: when information is superseded, delete the old version rather than keeping both. In extreme cases, start a fresh window that restates the current corrected state of the job and leaves the poisoned material behind.
Key Takeaway
The four failure modes and four fixes form a simple diagnostic framework. False conclusions call for validation, bloated history calls for compaction, excess tools call for curation, and contradictions call for pruning. The unifying lesson is that every fix works by discarding something, so an agent becomes better at long-running tasks by holding less. The video closes by inviting viewers to share their own context engineering strategies.