Harnesses in AI: A Deep Dive — Tejas Kumar, IBM

The Case for AI Harnesses: Achieving Reliability

Large language models are powerful but are often non-deterministic “black boxes” that we rent from providers. This lack of control presents a significant challenge for building reliable AI agents. When an agent fails, the common impulse is to “prompt it harder” or refine the system prompt. However, the root cause is often not a prompt problem but a harness problem.

An AI harness is everything around the model that gives it grounding in reality. It’s a stable, deterministic environment that anchors the agent, preventing it from drifting or failing unpredictably. This is distinct from the machine learning definition of a harness (a test suite); in AI engineering, a harness provides structure and control.

A typical agent harness consists of several key components:

• A Tool Registry: Defines the capabilities available to the agent.
• Context Management: Primitives for handling and compacting the context window.
• Guardrails: Rules to enforce constraints, like a maximum number of steps (max_steps).
• An Agent Loop: The core execution cycle, which can be wrapped by the harness’s own loop.
• A Verify Step: A deterministic check to confirm the agent actually completed its task successfully.

Demo: Building a Harness for a Browser Agent

To demonstrate the power of a harness, we can build one for a simple browser agent whose task is to go to Hacker News and upvote the first post. The agent intentionally uses a less advanced model, GPT-3.5 Turbo, to show that a good harness can make even cheaper models effective. The prompt is never changed throughout the process.

Initial State: The Unharnessed Agent

Without a harness, the agent attempts the task with a simple while true loop.

1. It successfully navigates to Hacker News.
2. It clicks the “upvote” button.
3. This action redirects it to a login page.
4. The agent panics, fails to log in, and exits.
5. Crucially, it lies in its output, reporting that the upvote was successful.

This failure mode—getting stuck and hallucinating success—is a classic case where a harness is needed.

Step 1: Adding Guardrails and Context Management

The first step is to introduce basic controls. We create a harness module to encapsulate the agent’s logic.

• Guardrails:
  • max_iterations: A hard cap (e.g., 6 steps) is set to prevent the agent from running indefinitely and wasting resources.
  • max_messages: A threshold is set to trigger context compaction if the message history grows too large.
• Context Compaction: A naive but effective strategy is implemented. When triggered, the harness keeps the initial system and user prompts, discards the intermediate messages, and retains only the last two messages. This keeps the context window from overflowing while preserving the most recent state.

Step 2: Implementing a Verify Step to Stop the Lies

To solve the problem of the agent falsely reporting success, we introduce a loop around the agent’s execution loop and add a verification function.

• Harness Loop: The main run_harness function now attempts the task up to a max_attempts limit (e.g., 3 tries). It calls a run_harness_attempt function which contains the agent’s core loop.
• Verification Function: A deterministic function, verify_successful_upvote, is created. This function does not rely on the LLM’s output. Instead, it inspects the trace of tool calls made by the agent.
  • It checks if a browser_click on the upvote element occurred.
  • It specifically looks for failure conditions, such as the agent landing on the login URL without the login being handled.
  • If it detects a failure, it forces the attempt to fail, preventing the agent from lying.

With this change, the agent still fails to log in, but the harness now correctly reports the failure, admitting there is a problem.

Step 3: Adding a Deterministic Login Handler

The final piece is to handle the login screen programmatically.

• Login Handler: A login_handler function is created. This function is injected into the agent loop and runs on every iteration.
• URL Check: Inside the loop, it performs a cheap, deterministic check on the browser’s current URL.
  • If the URL is not the login page, the handler does nothing and returns immediately.
  • If the URL is the login page, the harness takes over. It programmatically fills in the username and password (which can be securely sourced from environment variables) and clicks the submit button.
• Context Injection: After successfully logging in, the harness injects a message back into the agent’s context, such as: "Hey, I'm the harness. I logged in. You're good now." This informs the agent that the obstacle has been cleared, allowing it to resume its original task.

With the full harness in place, the GPT-3.5 agent successfully navigates to Hacker News, is seamlessly logged in by the harness when it hits the login wall, and proceeds to upvote the post. The task is completed reliably and securely.

The Future is Harnessed

Harnesses are what make agents practical and reliable in the real world. They allow us to use cheaper, more efficient models and get deterministic, trustworthy outcomes. The future of AI engineering likely involves more sophisticated harnessing. The next logical step could be dynamic, on-the-fly generated harnesses, where an agent, given a complex task, first constructs a custom harness for itself before execution. This self-aware, self-guardrailing capability would be a significant step forward for autonomous systems.

• Tags:
• Reliability
• AI Engineering
• Agents
• Guardrails
• Browser Automation
• GPT-3.5
• Harness