Overcoming Agentic Memory Management Challenges
Biswaroop Bhattacharjee from Prem AI discusses Cortex, a novel AI memory system inspired by human cognition. The conversation explores moving beyond traditional flat memory structures to hierarchical, context-aware systems that enable more sophisticated and less noisy retrieval for AI agents.
The current exploration in AI memory challenges us to think beyond simple data storage and retrieval, moving towards systems that mimic the complex, interconnected nature of human memory. This involves creating dynamic structures that understand relationships, context, and hierarchy, much like our own minds.
Inspiration from Human Cognition and AMM
The journey into building a more sophisticated memory system, Cortex, began with inspiration from the Agentic Memory Model (AMM) paper. A key differentiator in AMM is its ability to form relationships between memory entries. When a new piece of information is added, it’s not just stored in isolation. The system performs a semantic search to find related existing memories and then uses an LLM to identify and establish connections between the new entry and the old ones.
This process is analogous to how humans connect new experiences with past memories. For instance, this current conversation is connected to the memory of scheduling it. AMM creates a knowledge graph-like structure where each memory “chunk” is enriched with metadata (summaries, keywords, tags), and these chunks are interconnected, preventing the information over-compression often seen in traditional graph RAGs.
Cortex: Evolving Memory with Scored, Bidirectional Relationships
Cortex builds upon the foundation of AMM by introducing more nuance to the relationships between memories. The key innovations include:
- Named and Scored Relationships: Instead of just connecting memories, Cortex assigns a name (e.g., “extends by,” “is a definition of”) and a confidence score to each relationship. An LLM determines the relationship type and its strength, creating a more meaningful and weighted graph.
- Bidirectional Connections: Relationships are not one-way. If memory A “defines” memory B, then memory B “is defined by” memory A. This allows for more comprehensive context retrieval, as traversing the graph from any node reveals a richer set of connections. This is similar to setting the depth in graph traversal.
- Memory Merging: Cortex experiments with merging memories, similar to how humans sometimes consolidate or conflate different events. This can be a simple concatenation or a more complex update, with the original history maintained in the metadata in case they need to be separated later.
Hierarchical Structures: Smart Auto-Collections
A significant leap in Cortex is the move from a “flat” memory structure, like that seen in tools like Obsidian, to a hierarchical one. Flat structures are powerful for connections but lack the organizational depth of a file system. Humans naturally organize information into hierarchical topics and subtopics (e.g., Work -> Project A -> Task 1).
Cortex replicates this with “Smart Auto-Collections”:
- Categorization: When a new memory is inserted, an LLM call assigns it categories and subcategories (e.g., “Work,” “Python,” “Emotions”).
- Collection Formation: When the number of memories in a category reaches a certain threshold, a “collection” is automatically formed in the background.
- Collection Metadata: This new collection is given its own metadata, including a summary/description and a “query helper”—a meta-prompt that guides the LLM on how to best query the contents of that specific collection.
This hierarchical approach enables a more focused, higher-quality search, significantly reducing noise. A vague query like “fix this” can be contextualized by first identifying the most relevant high-level collection (e.g., “Work”) and then searching within that narrowed-down context.
Advanced Hybrid Retrieval
To leverage both the global connections and the new hierarchical structure, Cortex employs a hybrid retrieval mechanism that combines two parallel searches:
- Global Search: The original method of searching the entire “flat” knowledge graph.
- Auto-Collection Search: A constrained search that first identifies the most relevant collections and then searches within them.
The final results are ranked using a composite score, giving different weights to the collection’s relevance and the individual memory’s relevance within that collection. For example, collection similarity might get a 30% weight, while the intra-collection search gets a 70% weight, ensuring that the most contextually appropriate items are surfaced.
Handling Temporal and Contextual Queries
Cortex also addresses time-based queries. By recognizing keywords like “last week” or “today,” the system can apply a “temporal weight.” This weight adjusts the search parameters, biasing the retrieval towards more recent memories stored in a short-term memory window or expanding the search within a specific date range. This allows an agent to answer questions like, “What did I talk about yesterday?” with much higher accuracy.
Memory as a Tool for Agentic Systems
Ultimately, the vision for Cortex is not as a user-facing tool with manual parameters but as a sophisticated tool for an AI agent. In the paradigm of context engineering, an agent can be given Cortex as its sole memory tool. The agent itself would learn to tweak the parameters—adjusting the temporal weight, deciding whether to use global search, or setting the retrieval limit (K)—based on the specific task. This approach streamlines the agent’s toolset, reducing the cognitive load and potential for error that comes with managing numerous, overlapping tools, which has been shown to decrease task completion rates.
The Future: Multi-Modal Memory
The next frontier is expanding beyond text to include multi-modal memory, particularly vision. The current method of compressing video or audio into a textual description is lossy. The goal is to develop methods for indexing and storing vision and other sensory data in its native, high-context format (likely as vectors), allowing for direct, cross-modal communication. A future agent’s vision memory should be able to seamlessly interact with its textual memory, creating a truly comprehensive and human-like understanding of the world.