Hierarchical Memory: Context Management in Agents — Sally-Ann Delucia

The focus in AI development has shifted from prompt engineering to the more critical challenge of context engineering. While prompts were the initial focus, experience shows that the context provided to a model is what ultimately determines an agent’s success or failure. The best context strategy is one that allows an agent to remember what it needs to and forget what it doesn’t.

The Vicious Loop: An Agent Analyzing Itself

When building Alyx, the Arize AI agent, the team encountered a unique and challenging feedback loop. Alyx is designed to operate on its own trace and span data to help developers understand their AI applications. This created a vicious cycle:

1. Alyx runs on trace and span data.
2. The spans grow as the agent operates.
3. The growing data eventually exceeds the context limit.
4. Alyx fails, and this failure is recorded in the spans, adding even more data to the context.
5. The agent tries again with an even larger context, leading to another failure.

This feedback loop made effective context management a non-negotiable requirement. It’s not just an engineering problem but a core product and UX issue—bad context leads to bad answers, which ruins the user experience.

Escaping the Loop: An Evolution of Strategies

The team iterated through several strategies to manage the context window, with clear lessons learned at each stage.

Attempt 1: Naive Truncation

The first, most straightforward approach was to simply truncate the context, keeping only the first 100 characters.

• Outcome: This failed immediately. The agent effectively had no memory, treating follow-up questions as entirely new conversations. The key takeaway was that over-truncation completely breaks the agent’s reasoning ability.

Attempt 2: Summarization

The next logical step was to use an LLM to summarize the entire context into a more compact form.

• Outcome: This approach proved too inconsistent and unreliable. Leaving the summarization to the LLM meant giving up control over what information was deemed important. This lack of control made the agent’s behavior unpredictable.

The Solution: Smart Truncation + Memory Store

The strategy that ultimately proved successful combines intelligent truncation with a retrievable memory system. This is the approach Alyx uses today.

• Mechanism:
  • Preserve the head (first 100 characters) and the tail (last 100 characters) of the conversation context.
  • The middle of the conversation, which often contains verbose but less immediately relevant data like tool calls, is truncated.
  • This truncated middle section is stored in a separate, retrievable memory store.
  • The agent is given the ability to query this memory store if it determines that information from the middle of the conversation is needed to answer a question.

This method provides a balance, keeping the immediate context lean while ensuring the agent can access historical information when necessary. It embodies the principle: “Context decides what the model sees, memory decides what survives.”

Broader Architectural Lessons

Beyond managing a single context window, the team learned two other crucial lessons for building robust agents.

1. Long Session Evals

Users tend to have long, continuous chat sessions. A context management strategy might work for short conversations but fail after many turns. These late-stage failures are difficult to catch.

• Solution: The team implemented long session evaluations. They programmatically load a conversation with 10 turns of history and then test the agent’s performance on the 11th turn. This makes late-stage degradation a testable and fixable issue, rather than something discovered by users in production.

2. Offloading to Sub-Agents

The team realized that not all context belongs in the same agent. For complex, data-heavy tasks like searching over hundreds of spans, loading all the intermediate reasoning and data into the main chat context was inefficient.

• Solution: They adopted a sub-agent architecture.
  • The Main Agent handles the user-facing conversation and maintains only a “light” context (chat history).
  • For heavy tasks, it delegates to a specialized Sub-Agent.
  • The Sub-Agent operates in its own context, loaded with all the heavy data required for the task. It performs its function and passes only the final, concise result back to the Main Agent.

This approach keeps the primary conversation responsive while enabling the system to perform complex, data-intensive operations.

Ongoing Challenges and Key Takeaways

The work is far from over. The team is still tackling several challenges:

• Long-Term Memory: The current memory store is session-based. The next step is building true long-term memory so Alyx can reference issues from previous, separate conversations.
• Context Selection: The “head/tail” truncation is still a heuristic. The team is researching more principled methods for defining a “context budget” and metrics for context quality.
• Industry Parallels: A look at the released Claude source code revealed that they use a similar truncation and compression strategy, suggesting the industry is converging on these types of solutions while still searching for a silver bullet.

Agents don’t fail because of prompts, they fail because of context.

The most important takeaways from this year of development are that Context Engineering, Memory, and Evaluation are the three pillars of building a successful AI agent.

• Tags:
• AI Agents
• LLM
• Memory Management
• Context Window
• Truncation
• Summarization
• Sub-Agents