When to Build Your Own Agent Harness | Harrison Chase, LangChain

Harrison Chase, co-founder and CEO of LangChain, discusses the critical role of the ‘harness’ in LLM agents, emphasizing its function in orchestrating models and context for effective intelligence. He outlines that true ownership of intelligence necessitates owning all three components: the model, the context, and the harness.

The Agent Triad: Harness, Model, and Context

An agent, at its core, comprises three main parts: a harness that orchestrates a model (like an LLM) and relevant context. Owning one’s intelligence implies owning all three. While model ownership involves selecting the best available model and maintaining flexibility to switch, and context ownership includes managing memory, semantic knowledge, and conversation history, the primary focus is on the harness.

The Harness: Orchestration and Context Management

The harness’s fundamental job is to “bring context to the model at the right point in time.” It orchestrates the flow of fixed and dynamic context into the model’s context window, processes the model’s response, and manages interactions with external systems. These external interactions often generate new context that feeds back into the agent’s loop.

The core architecture of most agents involves an LLM running in a loop: a request comes in, the LLM generates a response (potentially including a tool call), the tool is invoked, and its observation is passed back to the LLM. This simple, yet general, loop is the foundation for all agents.

Customizing the Core Loop with Middleware and Architectures

While the core loop is simple, harnesses can be extensively customized. LangChain provides a base harness, while Deep Agents (a more general-purpose version of Claude Code) adds capabilities like file system access, skills, and sub-agents. Customization occurs through:

• Middleware/Hooks: Code snippets or plugins that wrap model calls and tool calls, allowing modifications before or after their execution. Examples include adding sandboxes, file system access, sub-agents, memory management, and summarization (to shorten context before model invocation) or context offloading (for large tool outputs).
• Cognitive Architectures: For highly specific or complex flows, explicit cognitive architectures (like those for deep research or code review bots) were historically used when models were less capable. While much functionality has been absorbed into general harnesses, these bespoke architectures remain valuable for guiding agents in precise ways, especially when predictability and control are paramount (e.g., in financial services). It’s recommended to start with a general harness and then add specific gates and checks as the use case demands.

Build vs. Buy: When to Customize Your Harness

The decision to build a custom harness versus using an off-the-shelf solution (like Claude Code or Codex) hinges on the task’s distribution:

• In-Distribution Tasks: For tasks aligned with how models are pre-trained, off-the-shelf harnesses are often sufficient and perform well.
• Out-of-Distribution Tasks: As tasks move further from the models’ training distribution, customization becomes necessary. For example, while legal AI might be out-of-distribution, specific sub-tasks like file editing might be in-distribution for models, but with different optimal approaches depending on the model (OpenAI vs. Anthropic). Custom harnesses, like Deep Agents, can use model profiles to switch between model-specific implementations for these in-distribution components, ensuring efficiency while maintaining overall task customization.

Evals and Observability: Driving Continuous Improvement

Experimenting with agent components requires robust evaluation (evals) and observability. As highlighted by Satya, “eval defines what good looks like inside the organization,” and retaining ownership of traces and feedback creates a continuous learning loop.

• Evals and Benchmarks: Companies building mission-critical agents must create benchmarks to define success, catch regressions, and improve agents (by adjusting harness or model). Harbor is an open-source eval runner for this purpose, allowing agents to be run against datasets of tasks within sandboxed environments. Each Harbor task defines an environment (via Dockerfile), a solution, a test script (for verification, using code or even an LLM/agent as a judge), and instructions (the prompt). Benchmarks allow comparison of different harnesses and models based on accuracy, latency, and cost.
• Observability: Often underrated, observability is crucial for debugging agents. Most agent failures stem from issues with the context provided to the LLM, not necessarily the model’s inherent capability. Detailed observability into the context window, tool calls, and overall execution traces (represented as trajectories or message lists) helps diagnose and resolve problems.

The Data Flywheel: Traces to Intelligence Compound

The data flywheel describes a continuous process for improving agents:

1. Run Agent & Collect Traces: Deploy the agent and gather execution traces.
2. Curate Trace Data: Analyze traces and collect feedback.
   • Feedback Mechanisms: This can be implicit, through clever UX design that surfaces user intent, or synthetic, using online evaluators (e.g., fine-tuned SLMs or custom-prompted LLMs) to judge agent performance efficiently.
3. Run Experiments & Update Agent: Use the curated data to run experiments and update any part of the agent:
   • Harness Engineering: Improve the orchestration logic.
   • Model Fine-tuning: Enhance the model’s performance on specific tasks.
   • Context Updates: Refine memory or knowledge retrieval.

LangSmith Engine Demo: Automating the Flywheel

LangSmith Engine is an agent designed to automate this data flywheel. It analyzes traces, identifies common issues, curates them into ‘issue boards,’ and suggests fixes (primarily focusing on harness engineering). An example of its effectiveness is the “codexification” sprint, where observations from Codex’s performance on a benchmark led to integrating its script-writing behavior into the Engine harness.

Harness Convergence vs. Divergence

The future of harnesses presents a question of convergence or divergence. While general-purpose harnesses are becoming good enough for many basic tasks, the need for customization grows with out-of-distribution tasks. Factors like requirements for predictability, control, and even subtle differences in how model labs train for specific sub-tasks (e.g., file editing) suggest that while some convergence may occur for common functions, significant divergence will likely persist for specialized domains or those requiring bespoke cognitive architectures. Evals and observability remain key to navigating this evolving landscape.

• Tags:
• Feature
• Observability
• Context Management
• Benchmarking
• Data Flywheel
• LLM Agents
• Middleware
• AI Development
• LangChain
• Agent Orchestration
• Agent Evals
• Machine Learning Engineering
• AI Harnesses
• Cognitive Architectures
• Harbor
• LangSmith Engine