Turning Agent Memory Into Skills That Work — Will Lyon, Neo4j
Neo4j's Will Lyon argues that embedding-based retrieval gives agents recall but not actionable knowledge. He presents a context graph model of agent memory with short-term, long-term, and reasoning memory, then shows how to distill grounded, governed skills from that graph using APE typed execution graphs, with grounding, coverage, and coherence checks, staleness alerts, and a live demo of the Neo4j Agent Memory Service building a patient-intake skill.
Why retrieval alone is not enough
The talk starts from a problem many agent builders will recognize: agents work through reasoning and acting loops, complete a task, and then forget what they learned. The speaker, a senior product manager at Neo4j, argues that the dominant memory pattern of embedding text and retrieving similar chunks at query time solves recall but not actionable knowledge. Retrieved chunks are injected into the context window and the agent is trusted to do something useful with them. The core gap is the lack of a canonical representation of things. The same cardiologist may appear as Dr. Nwin, Robert N, or simply ’the cardiologist’ depending on context, and a retrieval system has no native way to know these all refer to one provider with a known name, role, and set of relationships. The thesis is that agents need knowledge that is connected, typed, and traceable. Retrieval is only one part of the problem.
The context graph model of agent memory
Neo4j frames agent memory as a single connected graph made of three parts: short-term memory, long-term memory, and reasoning memory. Short-term and long-term memory cover the conversation itself and the knowledge extracted from it. The pipeline starts with user and assistant messages, runs entity extraction to identify the entities and how they relate, and writes them into a graph with explicit types and relationships.
Two mechanisms do most of the work here. The first is entity resolution, which the speaker calls one of the most important steps in building the graph. It ensures that every mention maps to a canonical node. The second is a shared ontology, a description of the domain’s data model. In the healthcare example, the ontology defines encounters, providers, and similar classes, which gives extraction and resolution a fixed vocabulary to work against.
Reasoning memory and decision traces
The more distinctive claim is that reasoning should be a first-class citizen in the graph. Messages and extracted entities describe what was said, but not how the agent reached its decisions. The proposed answer is decision traces. Each decision is linked to the reasoning behind it and grounded in evidence. The trace records the policies being applied, the execution plan, the tools called and their results, and operational metrics such as token usage and timing. The point is to store the facts of how the agent thought, not only the data it touched.
The payoff is collective. Stored traces can be shared across a fleet of agents, potentially hundreds or thousands, that have access to a common set of tools. Instead of each agent improving only itself, the organization gets one shared context graph in which agents learn from each other’s runs.
From memory to skills
The talk then moves from memory to action. Memory provides a faithful record of what happened, but the next step is usually a skill. The speaker describes the open Agent Skills standard originally released by Anthropic. A skill has metadata describing its purpose and progressively disclosed detail, often in Markdown reference files that the agent loads only when needed. This makes skills a portable, reusable unit.
The weakness is that skills written as prose share the same problems as unstructured memory. It is hard to pin down what the canonical entities are, and the steps are hard to debug or verify against outputs.
Skills as typed execution graphs
To address this, researchers on the Neo4j team published work on APE, a graph representation for learning and governing agent skills. APE extends the Agent Skills protocol with additional metadata that models a skill as a typed execution graph. Steps become nodes, and each node is governed by a strict schema describing how the step is executed. The speaker reports that on the SkillsBench benchmark, applying the APE representation to human-curated skills produced a significant increase in successfully completed tasks.
Distilling grounded, governed skills
The proposed pipeline distills a skill from a subgraph of the context graph, scoped to a workspace, project, entity, class in the ontology, or a single conversation. The requirements are that the skill be grounded in observed data, deterministic in its procedural steps, and governed over time as the underlying data changes.
Three heuristics guide the process:
- Grounding: each component of the skill must trace back to data in the memory system.
- Coverage: steps and descriptions across the skill should be supported by grounded evidence.
- Coherence: if the source subgraph spans multiple topics, the system should recommend splitting it into several skills. Community detection algorithms help identify these topic boundaries.
Governance closes the loop. If the data behind a skill changes, is removed, or becomes contradictory, the skill is flagged as stale or drifted. A governed skills registry can then serve each agent the most recent up-to-date version. The speaker stresses that most pipeline stages are deterministic. An LLM is used only to synthesize some claims and generate parts of the descriptive text.
The Neo4j Agent Memory Service
These ideas are implemented in the Neo4j Agent Memory Service (NAMS), an agent-memory-as-a-service offering from Neo4j Labs, which is currently free to sign in and try. Each workspace has a scoped context graph containing the three memory types. Background workers run the distillation process, and a curation process monitors for staleness. The service exposes REST and MCP tools for ingesting and retrieving short-term, long-term, and reasoning memory. Open-source tooling implementing the patterns is also available.
Demo: from a healthcare conversation to a patient-intake skill
The demo runs in the NAMS dashboard against a healthcare ontology and a set of ingested conversations with a healthcare agent. The presenter chose the most recent conversation as the distillation scope and started a job that runs through the seven-stage pipeline and constructs the skill as a graph.
A previously completed run was shown as an example. It was distilled from a patient-intake conversation and captured the full sequence of steps, from intake through charting. Each step was grounded in the tool calls and entities that produced it. The result was packaged as a SKILL.md file, with references arranged for progressive disclosure as in the Agent Skills standard.
Takeaway
The talk’s central claim is that agent memory should be a typed, connected, and traceable graph that records both facts and reasoning. Skills should be distilled from that graph as governed execution graphs rather than hand-written prose, so that they stay grounded, verifiable, and current as the underlying knowledge changes.