Memory-Aware Agent Harnesses for Anti-Money Laundering — Nacho Martinez, Oracle
Nacho Martinez argues that the model is fixed while the harness (agent loop, memory, context engineering, tools and skills, continual learning) is where engineers can make real gains. He walks through the agent stack and applies it to an autonomous AML agent that flags suspicious bank transactions, then demonstrates the workshop environment and answers questions on memory design, skill retrieval, caching, and hybrid search.
The speaker opens with a thesis that organizes the whole session: the model is the part you cannot change, so the real gains come from the harness around it. He frames the talk as roughly 15 minutes of theory followed by a hands-on workshop, where attendees build an autonomous anti-money-laundering (AML) agent for a fictional mid-sized bank.
The use case: an AML agent for a bank
The workshop is adapted to a bank audience. The bank receives many transactions, and some are suspicious. He gives the example of a $1 million deposit followed by repeated $50,000 ATM withdrawals over three days. Suspicious transactions fall into five categories. The goal is an agent that works around the clock, flags or decides on these transactions, and reduces the time human analysts spend on discovery. He stresses that the agent should not block accounts or ruin lives on its own. It should print its decisions so a human can inspect them later.
The agent stack and the data layer
He describes a stack with four layers: the application layer (chatbot, mobile app, or interface), the model (the reasoning core, over which users have little say), the infrastructure, and the compute. The session focuses on the data layer, since that is where engineers have real control. Within it sit the agent loop, a gateway and MCP for third-party data sources, the memory layer, context engineering (summarization and compaction), and tools and skills. The central idea is to keep the context window small through these components.
He distinguishes four common application shapes: LLM chatbots, RAG (which he calls a glorified vector search), LLM-driven workflows, and AI agents. Tools like Claude Code, Codex, and OpenClaw combine workflows, which provide automation, with agents, which provide autonomy. He argues that most engineers use such agentic systems daily but rarely design their own applications this way.
What an agent is, and why the harness matters
He defines an AI agent as a computational entity with a reasoning core (the LLM), memory (a database), tools (its way of acting on the world, like a human’s extremities), and grounded inputs (perception, like human senses). He also notes that attention is limited for both humans and agents. The harness is the scaffolding that implements the tool use, memory, and perception. He claims the same frontier model can perform noticeably better or worse depending only on its harness, and that many teams underinvest in harness design.
He lists the decisions a harness requires: which model to use, where data is stored, how vectors are compared and retrieved, how memory is divided, and how to represent the semantic layer, meaning the shared unstated context that humans and models both rely on.
Files versus databases
He raises the debate over whether agents should store state in files or databases. Using multiple agents writing to one file as the example, he shows that the current workaround, git worktrees, exists because files lack transactional guarantees. Concurrent writes raise consistency questions such as which write came first and what a reader sees. Databases solved these problems decades ago through ACID properties (atomicity, consistency, isolation, durability). His position is that files suit unstructured data and databases suit structured, transactional state, and that engineers should not reinvent solved problems.
Memory and context engineering
He divides agent memory into short-term and long-term. The context window is short-term memory and vanishes after a conversation. Long-term memory has three categories: episodic memory (past conversations, tool logs, and summaries), procedural memory (skills and workflows), and semantic memory (user preferences and facts the agent should know).
He makes a key empirical point: performance degrades as the context window grows, even well below advertised limits. A context that starts near zero tokens solves problems more accurately than one already holding 256,000 tokens. Two strategies address this. Context summarization asks the model to condense the conversation and continue from the summary. Compaction stores chunks of the conversation in a database or file with an ID, and the agent later retrieves them by ID and offset, much like reading a file.
The agent loop
The agent loop drives everything. It assembles context dynamically by pulling skills, tools, prior conversations, files, and database records into the window. It then calls the model API to reason and decide the next action, and repeats. In the workshop, the loop is capped at eight iterations and 60 seconds of reasoning per iteration, which he describes as configurable and a deliberate guard against runaway costs. He emphasizes that engineers decide what goes into the loop.
Toolbox and skillbox patterns, skill distillation
With hundreds of installed skills, each several thousand tokens long, loading them all would exhaust the window. His proposed pattern stores tools and skills as vector embeddings in a toolbox and a skillbox. The agent loop runs a similarity search to retrieve only the most relevant ones and offloads them after use. He notes that tools and skills can consume more than half of the context window, so engineers should be deliberate about them.
He then describes skill distillation: converting past runs and conversations into a skill.md file stored in the skillbox. He cites Hermes Agent as an example of this approach. For instance, a front-end developer’s preferred coloring style, once distilled into a skill, can be retrieved semantically later without being named explicitly. He frames this as dynamic skill discovery and a practical route to learning over time.
How a frozen model gets better: continual learning
Since frontier models are fixed from the user’s perspective, he argues that improvement must come from the data layer. He sorts approaches into three categories: parametric memory (changing weights, for example through fine-tuning or LoRA), representational memory (embeddings, including custom embedding models), and nonparametric memory (everything discussed so far, such as external stores, retrieval, and skills). He focuses on the third and points to a free DeepLearning.AI course with Andrew Ng for the other two.
Instruction hierarchy
He explains that model APIs weight inputs differently. The system prompt carries the most weight, then the user prompt, and tools and skills carry the least. Deliberately placing instructions across these levels helps the model focus on what matters. He treats skills as a plug-and-play layer and describes ongoing experiments to connect external skill hubs and MCP tool collections through precomputed embeddings.
Workshop demo
He demonstrates the workshop environment on GitHub Codespaces. The fake transaction data stream includes both legitimate and problematic transactions. An autonomous agent running a Grok model on Oracle Cloud Infrastructure analyzes them, calls its tools, searches knowledge spaces built on vector stores, and writes suspicious activity reports. The interface also shows the agent’s context window, the database contents, and the 30 most recent bank events. Participants work through a student notebook.
Q&A highlights
On episodic memory and conflicts, he says the answer depends on the architecture. Keeping memory layers cleanly separated avoids confusion, and important episodic memory can be vectorized rather than stored as raw text.
On caching and budgets, he explains that around 80 percent of a typical request is tools and skills, which providers cache semantically. Without that cache, costs would be four or five times higher. He compares harnesses by their initial token overhead: Pi is minimal at about 6,000 tokens, while others start near 30,000 to 40,000. In the workshop, context assembly retrieves the top three tools and top three skills, and a reranker could further narrow them.
On memory simplicity, he recommends starting with only short-term and long-term memory. Information gets promoted from the window to long-term storage as raw text, then vectorized and recalled through an added recall step between observing and reasoning.
On SQL and MCP, he argues that MCP’s main value for databases is offering pre-written, proven SQL queries. This avoids hallucinated queries, since SQL dialects such as PostgreSQL and Oracle differ in subtle ways.
On files versus embeddings for skills, he says regex search over files scales poorly, while embeddings capture meaning. He recommends hybrid search combining semantic and keyword retrieval, using reciprocal rank fusion (RRF), which he says improves quality at the cost of extra compute.
Build small custom harnesses
His closing advice is not to rebuild Claude Code, but to build small, custom harnesses for specific problems and start from scratch to strip out what is unnecessary. A consultant, for example, could maintain a financial harness and adapt it to new clients’ constraints. Different problems need different designs. Autonomous math agents, for instance, need automatic compaction and near-infinite patience, while an AML agent needs a bounded loop. All these harnesses sit on the chat completions API, and he argues that the smaller the harness, the better it tends to perform.