Total Recall: Agent Memory and Harness Engineering — Ignacio Martinez, Oracle

Introduction to Agent Harness Engineering

Ignacio Martinez’s workshop introduces agent harness engineering, positing that while large language models (LLMs) represent the “frozen part of reasoning” (rented, immutable, and beyond direct control), the surrounding harness is the critical layer where engineers gain control. This harness is engineered to transform non-deterministic LLM outputs into reliable and predictable outcomes. The core objective is to equip participants with the knowledge to build their own agent harnesses and understand agent memory.

The Agent Stack and Data Layer Centrality

Martinez outlines a five-layer agent stack:

• Application (product surface)
• Data (memory, knowledge, retrieval, encoding, search)
• Model (LLM reasoning)
• Infrastructure (orchestration)
• Compute (cloud, GPUs, database engine)

He contends that Application, Model, Infrastructure, and Compute layers are increasingly commoditized, leaving the Data layer as the primary domain for engineering control. This layer is where the agent harness manifests, incorporating components like gateways/MCPs (connecting LLMs to external data and tools), memory, semantic, retrieval, context layers, and various tools/skills.

Defining an AI Agent and Harness Engineering Goals

An AI agent is precisely defined as a model plus a harness. The model serves as the reasoning engine, while the harness encompasses:

• Memory: Stored in databases or files.
• Tools: For performing actions.
• Perception: Grounded by environmental inputs.

The fundamental goal of harness engineering is to impose reliability and repeatability on the inherently non-deterministic nature of LLMs. This is achieved by combining automation (for reliability) with autonomy (for flexibility). The LLM itself is treated as a swappable component, emphasizing that the harness is the primary area of customizability and engineering effort.

The Seven Layers of an Agent Harness

Martinez meticulously details seven crucial layers (excluding the model layer) that constitute a robust agent harness:

1. Storage Layer: Files vs. Databases

This layer determines where an agent’s memory physically resides, addressing the ongoing debate:

• Files: Advantages include ease of creation, unstructured nature, alignment with model instincts, and POSIX semantics. Disadvantages are a lack of transactional consistency (requiring workarounds like “work trees” for parallel agents), absence of hybrid search, and no native backup mechanisms.
• Databases: Offer ACID consistency, high availability, native vector search, and robust backup solutions.
• Hybrid Approach: The recommended solution involves leveraging technologies like Oracle DBFS (Database File System) to store files directly within a database. This combines the file-like accessibility with database benefits (transactional consistency, vector search). A common strategy is to use files for short-term memory and promote critical information (e.g., user preferences) to structured database tables for long-term storage.

2. Memory Engineering: Encoding, Search, and Retrieval

This layer focuses on how data is managed for retrieval:

• Tools: Integration via langchain-oracle-db for seamless vector store operations.
• In-Database Embeddings: A key feature allowing embedding models to run directly inside the database. This enhances data security and retention by avoiding external API calls during the embedding process, crucial for enterprise environments.
• RAG Pipeline Overview: The process involves tokenizing documents, creating embeddings, storing them in vector stores, comparing user query embeddings, and re-ranking results with cross-encoders.
• Addressing Data Sprawl: RAG pipelines often generate diverse data types (text, JSON metadata, vectors) leading to a need for multiple specialized databases and significant data synchronization overhead. Oracle’s “converged database” solution aims to consolidate all data types (relational, JSON, spatial, graph) into a single database engine, simplifying data management, reducing complexity, and centralizing security.

3. Agent Memory: Retention, Reuse, Refinement, and Recall

Agent memory is the system enabling an agent to manage information effectively:

• Memory Types:
  • Short-term memory: Ephemeral, for immediate context (e.g., a coding agent’s current to-do list).
  • Long-term memory: Stores persistent knowledge, including episodic (past conversations), procedural (successful workflows), and semantic (facts, concepts) memories.
  • Shared memory: Facilitates collaboration between sub-agents or multiple agents.
• Context Rot: A critical issue where an expanding context window leads to “context degradation.” Due to the quadratic scaling of attention matrices, attention to individual tokens diminishes over time. The goal is to maintain a compact and salient context window.
• Oracle Agent Memory Package (OAMP): A managed solution designed to simplify the complexities of memory engineering (context compaction, summarization, information extraction). OAMP generates a “context card”, a structured representation of context that includes: topics, a summary of the thread, relevant information (facts, preferences, memories), episodic memories (unanswered questions), and recent messages for local context.

4. Semantic Layer: The Agent’s Umwelt

This layer captures the implicit meanings and shared understanding within an agent’s operational domain:

• Umwelt Concept: Borrowed from biologist Jakob von Uexküll, Umwelt denotes an organism’s perceived reality filtered through its unique sensory lens. For an agent, its “semantic lens” is shaped by its training data, institutional knowledge, and contextual inputs.
• Tribal/Institutional Knowledge: This layer embodies the unstated conventions, data models, query execution specifics, and metadata that are implicitly understood within an organization. It’s the background knowledge an agent leverages without explicit instruction in every interaction.

5. Agent Loops: The Driver of Autonomy

Agent loops are the core mechanism transforming a static model into an autonomous agent:

• Minimalist Loop: Typically follows an Observe -> Reason -> Act cycle.
• Failure Resistance: Designed to be resilient, ensuring the agent continually operates and adapts without exiting the loop, thereby providing genuine autonomy.

6. Context Engineering: Salience Management

Techniques focused on optimizing the relevance and size of the agent’s context window:

• Toolbox Pattern / Skillbox Pattern: Strategies for storing and retrieving available tools and skills. The goal is to load only necessary tools/skills into the context window precisely when they are required, minimizing context bloat.

Continual Learning and Skill Promotion

Martinez addresses how agents can evolve and improve beyond their initial, frozen model weights:

• Three Pillars of Continual Learning:
  • Weights: Expensive and typically infeasible for most organizations.
  • Representations: Involves improving embeddings and re-ranking mechanisms.
  • Context Window/Token Space: The most accessible, cost-effective, and realistic method for most, focusing on how information is presented and utilized within the context.
• Skill Promotion / Workflow Promotion: A key concept where successful, time-intensive workflows (e.g., a 4-hour task) are distilled into more efficient, optimized versions. The original workflow is retired, and the improved version is adopted. This mechanism allows agents to adapt to user preferences, optimal strategies, and evolving requirements over time.

Workshop Highlights and Q&A

The workshop involves a practical session to build an agent harness step-by-step. Key insights from the Q&A include:

• Managing Thousands of Tools: Achievable using hierarchical navigable small world (HNSW) indexes within a database, ensuring efficient retrieval. LLM-enhanced tool descriptions can further improve separability.
• Hysteresis Variable: A configurable parameter within the harness that defines its patience with the model, such as a maximum number of tool calls (e.g., 8-12) before abandoning a task. This can be adjusted based on the underlying LLM’s performance.
• Model Routing: A recommended strategy for efficiency and performance, involving an orchestrator that routes specific queries to a mixture of small, expert models. This token-efficient approach leverages specialized models for different problem types.

• Tags:
• RAG
• Feature
• LLM Agents
• Semantic Layer
• Model Routing
• Continual Learning
• Context Window
• Harness Engineering
• Agent Memory
• Agent Harness
• Oracle DBFS
• In-Database Embeddings
• Context Rot
• OAMP
• Umwelt
• Agent Loops
• Toolbox Pattern
• Skill Promotion