CrabRAG: Why Automated Assistants Need Graph Memory, Not More Tokens — Stephen Chin, Neo4j
Stephen Chin demonstrates how current LLM agent memory systems, relying on markdown files or vector databases, lead to token inefficiency, hallucinations, and a lack of multi-hop reasoning. He showcases graph databases as a superior alternative, providing precise, explainable, and auditable answers for complex, large-scale problems through a live demo of a home lab digital twin.
Stephen Chin, from Neo4j, addresses a fundamental limitation in current autonomous agent architectures: their memory systems. He argues that traditional approaches, relying on simple markdown files or vector databases, are inefficient and prone to errors, especially when scaling or tackling complex, multi-hop reasoning tasks. Chin champions the integration of graph databases as a more effective and robust memory solution for agents.
The Shortcomings of Existing Agent Memory
Chin identifies significant issues with prevalent methods for managing agent memory:
Markdown File Memory
- Token Inefficiency: Agents typically store their memory and skills as plain markdown files. This forces them to load a vast amount of data—often “over 100,000 tokens per round”—in the hope of finding relevant information. This process is repetitive, token-intensive, and costly.
- Scalability Challenges: While this system might function at a small scale, it “doesn’t work at large scale” because the constant ingestion of potentially irrelevant data quickly exhausts context windows and resources.
- Suboptimal Skill Selection: Skills, also treated as markdown files, are difficult for agents to contextualize and select appropriately, leading to inefficient or incorrect tool usage. Chin uses the analogy of “poor boy crabd here is not going to get that clam. He just doesn’t have the open clamshell skill” to illustrate this.
Vector Database Memory
- Similarity vs. Actual Relationship: Vector databases excel at similarity searches based on embeddings. However, Chin emphasizes that “similarity in vector space is not the same as actual relationships.” This crucial distinction often results in factual inaccuracies and hallucinations.
- Inability for Multi-Hop Reasoning: For complex inquiries requiring “large multi-hop reasoning chains,” vector databases prove inadequate. They struggle to connect disparate facts logically, even when all the necessary information is present, making them unsuitable for intricate scenarios like digital twins or sophisticated domain-specific problem-solving.
- Lack of Precision: Responses from vector databases frequently include information that is merely “related in some way” rather than precisely relevant, leading to “silly mistakes” and less accurate answers.
The Graph Database Solution
Chin advocates for graph databases as the superior memory architecture for agents, capable of overcoming the inherent limitations of markdown and vector-only approaches.
- Structured for Connected Data: Graph databases naturally store entities (nodes), their connections (edges), and associated properties. This structure is ideal for representing and navigating complex, interconnected information efficiently.
- Hybrid Vector + Graph Approach: The proposed architecture combines vector search with graph traversal. Vector search is initially used to identify relevant “seed nodes,” which then serve as starting points for graph traversal to pinpoint precise, related information through nearest neighbors, followed by ranking.
- Precision, Explainability, and Auditability: Graphs offer significant advantages in these areas:
- Accurate/Precise: By tracing actual relationships, graphs deliver “very precise information.”
- Explainable: Developers can readily “look at the graph which got returned” to understand the reasoning and data paths leading to an answer.
- Auditable: The system allows for clear identification of “the part of the graph which resulted in that answer,” enhancing transparency, trust, and debugging capabilities.
- Accessibility: Chin notes that advanced LLMs like Claude can “write Cypher better than I can,” simplifying the process of building and querying graphs even for those without extensive graph expertise.
Live Demo: Vector vs. Graph in a Home Lab Digital Twin
To vividly demonstrate the performance disparity, Chin presented a live demo using a digital twin of his home lab. This digital twin was constructed from the exact same initial markdown facts, but two distinct agent memory systems were employed: one leveraging a vector database, and the other a graph database (specifically Cognite with a Neo4j backend).
Scenario 1: Identifying End-of-Life Software Exposed to the Internet
- Question: “WAN exposed end of life software?” (Are there outdated software versions exposed to the public internet?)
- Vector Agent Response: “Couldn’t find specific details, excluded by policy for more precise information, yada yada yada, source it separately.” This was a generic and unhelpful response, failing to provide actionable intelligence.
- Graph Agent Response: Precisely identified “Tinsterland” (Chin’s daughter’s Minecraft server) running “Debian Jessie,” an outdated operating system, and accurately flagged it as a security risk. The graph traversal—from blue seed nodes to gray one-hop traversals, highlighting green winning nodes—provided “very precise actionable information.”
Scenario 2: Discovering Exposed Management Ports
- Question: “Exposed 0.0.0.0 management ports?” (Are there any management ports openly accessible to the network?)
- Vector Agent Response: “Check services configuration expect PFSense rule.” The agent effectively delegated the investigative task back to the user.
- Graph Agent Response: Directly identified “haroxy” and “openVPN” as having exposed management ports by tracing their direct connections through the PFSense router node, delivering a “very precise answer.”
Chin personally benefited from the graph agent, which helped him uncover and subsequently patch actual security vulnerabilities within his home lab network, underscoring the practical utility of graph-based memory.
Conclusion and Resources
The demo convincingly illustrates that for “anything at at large scale where it doesn’t fit into the 1 million context window of the modern models,” a “better memory system than just throwing things in markdown files” is indispensable. Graph databases provide the crucial multi-hop reasoning capabilities required for complex environments, ranging from personal home labs to extensive enterprise data centers and sophisticated financial services systems.
Chin recommends two key resources for further learning and implementation:
- “GraphRAG: The Definitive Guide”: A comprehensive book co-authored by Chin, covering graph RAG, memory architecture, industry-specific use cases, and agent development.
- Neo4j GraphAcademy: A free online training platform offering courses on agent memory, context graphs, and other essential graph technologies.