Using LongMemEval to Improve Agent Memory

Sam Bhagwat of Mastra details their process for optimizing AI agent memory using the Long Mem Eval benchmark. He breaks down memory into subtasks like temporal reasoning and knowledge updates, and shares how targeted improvements—such as tailored templates, targeted data updates, and structured message formatting—led to state-of-the-art performance, emphasizing the importance of iterative evaluation.

August 25, 2025 4 min read

Using LongMemEval to Improve Agent Memory — source: https://www.youtube.com/watch?v=FTokJt1ioeg

The core of building effective AI agents lies in the iterative loop of understanding system performance and then optimizing it. This process was applied to enhance the memory layer of the Mastra agent framework, with the learnings encoded into the framework for broader use. The key tool for this process was the Long Mem Eval benchmark.

Deconstructing Agent Memory with the Long Mem Eval Benchmark

Agent memory can be defined as the compression of a queue of chat messages and the ability to search over them to retrieve the right information for the LLM’s context window. The Long Mem Eval benchmark, featuring 500 questions, breaks this capability down into five distinct subtasks:

Information Extraction: The most basic task, involving the retrieval of information stated by the user or assistant within a single session.
Multi-Session Reasoning: This involves compressing long-running chat histories to extract and synthesize information across multiple distinct sessions.
Temporal Reasoning: The ability to retrieve and present time-based events from the chat history accurately, allowing the LLM to reason correctly about sequences and durations (e.g., distinguishing an event that happened “today” from one “5 months ago”).
Knowledge Updates: Memory must be dynamic. This task tests the agent’s ability to update its stored information, such as user attributes, when new, overriding information is provided.
Handling Missing Information: A crucial capability where the agent must recognize and communicate when it does not possess the information required to answer a query.

Mastra’s Memory Implementation and Initial Results

Mastra agents primarily utilize two types of memory to address these challenges:

Semantic Recall: This works by embedding all chat messages and storing them in a vector database (like PG vector or Chroma). When a query is made, a search is performed to find the most relevant messages (top_k). To provide richer context, a message_range parameter is used to retrieve messages surrounding the primary match, similar to a grep -C command.
Working Memory: This stores specific, structured attributes about the user or conversation. The effectiveness of this memory is highly dependent on the template used, which defines the attributes to track. A template for a personal trainer AI would differ significantly from one for a legal research AI.

When the initial Mastra memory implementation was run on the Long Mem Eval benchmark, it scored 67%, falling short of the 72% state-of-the-art result. This prompted a focused effort to improve the implementation.

An Iterative Path to State-of-the-Art Performance

A series of targeted improvements, guided by the benchmark, led to significant gains:

Tailored Templates for Working Memory: The first step was to dynamically generate a tailored template for each benchmark question using an LLM. This made the working memory more accurate, but the overall impact was limited as semantic recall held more weight in the final score.
Targeted Working Memory Updates: The team identified an inefficiency where new information triggered a full rewrite of the working memory, sometimes introducing errors. They shifted to more targeted updates, overwriting only the specific part of the memory that needed changing. This change alone brought the overall accuracy up to the state-of-the-art level.
Fixing Timestamps for Temporal Reasoning: A subtle bug was discovered where incorrect dates were being passed into the context, confusing the LLM’s ability to perform temporal reasoning. Correcting the timestamps to reflect the true time of the messages led to another couple of percentage points of improvement on the benchmark.
Improving Data Structure: Instead of presenting messages as a flat list, the structure was enhanced. Messages were grouped by day, with timestamps for each message included. This organized data structure made it easier for the LLM to perform temporal and session-based reasoning, further improving performance.

Key Lessons and Future Directions

The results underscore that configuration matters. For instance, setting the semantic recall top_k parameter too low (e.g., 2) yields far worse results than a more reasonable value (e.g., 20).

The breakdown of the benchmark into categories was also vital. It showed that improvements to semantic recall did not significantly affect single-session tasks, reinforcing the need for granular evaluations to understand the precise impact of any changes.

The work is not complete. Future steps include implementing other memory techniques like reranking retrieved results and adding episodic memory to see how they influence benchmark performance. The ultimate takeaway is that continuous improvement comes from a tight feedback loop: build an evaluation scaffold, iterate relentlessly, and use the data to identify and fix the subtle bugs and structural inefficiencies that hinder performance.