OpenClaw's Memory Sucks and the fix is simple — Dhravya Shah, Supermemory
Dhravya Shah, founder of Super Memory, details the evolution of his company from a simple RAG-based consumer app to a sophisticated, open-source context infrastructure for AI, and introduces a novel hooks-based memory solution for OpenClaw.
Dhravya Shah of Super Memory explains that the project began as a consumer-focused bookmarking application, born out of the “Bspace” builder community. The initial architecture was a straightforward Retrieval-Augmented Generation (RAG) system built on a vector database. However, as the user base grew to over 100,000, he encountered significant scalability and cost challenges with existing vector database solutions. This led him to engineer a highly efficient, custom infrastructure on Cloudflare, famously running for just $5 a month.
When this infrastructure was open-sourced, it gained massive traction, shifting the company’s focus from the consumer app to the underlying “context infrastructure” that developers were eager to adopt.
Beyond RAG: A Modern Framework for AI Memory
Through consulting and building for various companies, Shah identified the limitations of simple RAG. He argues that for an AI to achieve “true user understanding,” its memory system must incorporate four key components:
- Knowledge Updates: The ability to invalidate stale information and build upon existing knowledge, moving beyond static vectors.
- Temporal Reasoning: Giving the agent an understanding of time to enable effective personalization (e.g., knowing an event happened “last week”).
- Forgetfulness: A mechanism to discard information that is no longer relevant.
- User Profiles: A small, persistent profile (~500 tokens) that is utilized on every turn. This profile captures both static facts (e.g., “user is a CEO”) and episodic information (e.g., “user recently moved into a new office”), enabling proactive and personalized responses.
To power this, Super Memory developed its own knowledge graph, deliberately avoiding the traditional “triplet” (subject-predicate-object) model, which Shah criticizes for being slow and requiring too many traversals to retrieve useful information.
Solving the OpenClaw Memory Problem
A significant portion of the discussion focuses on the widely reported issues with memory in AI agents like OpenClaw.
> The way OpenClaw… inherently relies on tools to search through these memory .md files that it prepares… sometimes it won’t decide to search, which is probably the biggest problem here.
Shah outlines the core flaws in OpenClaw’s default tool-based memory:
- Unreliable Activation: The agent often fails to trigger the search tool when it should.
- Static & Stale: The
.mdfiles are static, lacking mechanisms for updates, forgetfulness, or temporal reasoning. - Inefficient Discovery: The agent must search through files to find context, which is slow and cumbersome.
Super Memory’s solution replaces this with a hooks-based approach. In every interaction, a hook automatically and dynamically injects a small, relevant context of less than 2,000 tokens directly from the Super Memory graph. This ensures the information is always fresh and eliminates the agent’s need to decide whether to search.
Hybrid Mode: The Best of Both Worlds
To address edge cases where a user might ask about an insignificant detail not captured in the curated memory graph, Super Memory introduced a “Hybrid Mode.”
- It first attempts to retrieve structured memories from the graph.
- If no relevant memories exist, it falls back to a RAG-based search over raw text chunks.
- This ensures the agent almost never misses context, and the newly relevant information can then be integrated into the memory graph for future use.
The State of Memory Benchmarking
Shah provides a critical take on the current landscape of memory evaluation benchmarks:
- Memory Bench: Super Memory has open-sourced its own evaluation platform,
Memory Bench, to standardize the testing of memory systems on quality, latency, cost, and recall metrics. - Long-MEM: A good benchmark that tests for the right things, but its scoring has a direct correlation with the sheer number of memories extracted, which isn’t always practical or cost-effective in real-world applications. It also fails to test for forgetfulness.
- LocoMo: Criticized as “really bad” and outdated. It primarily tests retrieval capacity, and modern LLMs with long context windows can achieve near-perfect scores by simply “context dumping,” bypassing any real memory function.
- Convo-Mem: Praised as a good benchmark from Salesforce that tests for more realistic assistant use cases requiring a mix of RAG and memory.
A major gap in all current benchmarks is the inability to effectively evaluate the performance of user profiles and personalization.