Build Agents That Run for Hours (Without Losing the Plot) — Ash Prabaker & Andrew Wilson, Anthropic

The Evolving Challenge of Long-Running Agents

Building AI agents that can run reliably for hours or even days presents three core challenges:

1. Context: Finite context windows lead to amnesia between sessions. As a session progresses, agents suffer from “context rot” (losing coherence) and “context sense anxiety” (rushing to finish as the window fills).
2. Planning: Out of the box, models are poor planners. They might try to complete a complex task in one shot or build half-finished features before running out of context.
3. Self-Evaluation: Models are inherently bad at judging their own output. Due to sycophantic tendencies, an agent might mark a feature as “done” (e.g., a UI button) even if the backend functionality is completely missing.

Solutions to these problems come from two directions that co-evolve: improving the base models themselves and, crucially, building better “harnesses”—the scaffolding and logic that orchestrate the model’s work. Over the last year, models like Claude have progressed from running for an hour to over 12 hours on benchmark tasks with a minimal harness. However, the harness is what unlocks their full potential for complex, multi-hour application development.

The Adversarial Evaluator: A Better Approach to Quality

The common practice of having an agent check its own work is a trap. A more effective pattern is inspired by Generative Adversarial Networks (GANs), creating a system with two distinct roles:

• The Generator: An agent that writes the code and builds the application.
• The Evaluator: A separate agent whose sole purpose is to critique and test the generator’s output. It doesn’t just read code diffs; it uses tools like Playwright to open the live application, click around, and test functionality like a human QA engineer would.

This division of labor is effective because it exploits the gap between an LLM’s ability to create versus its ability to critique. It is far more tractable to tune a standalone critic to be harsh and objective than it is to make a builder agent adequately self-critical. It’s the same reason it’s easier for a person to critique a meal than to cook it.

A key benefit of this adversarial pressure is the ability to pivot. If a generator agent gets stuck and repeatedly fails to improve on a specific criterion (e.g., “originality”), the harness can discard the entire attempt and start from scratch—a course correction that a single agent trying to patch its own work will rarely make.

From Vague Prompts to Working Apps: The Three-Agent Loop

To build full-stack applications from a simple one-line prompt (e.g., “build a retro game maker”), the harness can be structured like a small software team:

1. The Planner (PM): Takes the initial prompt and breaks it down into a high-level spec with a series of sprints. Crucially, it avoids specifying granular technical details to prevent early architectural errors from cascading through the entire project.
2. The Generator (Engineer): The agent responsible for implementation.
3. The Evaluator (QA): The adversarial agent that tests the output.

The Glue: Negotiated Sprint Contracts

The most critical innovation in this structure is the interaction between the Generator and the Evaluator. Before any code is written for a sprint, the two agents negotiate and agree on what “done” means.

• The Generator proposes a feature and a set of verification tests.
• The Evaluator pushes back, perhaps arguing the scope is too large, the tests are too weak, or edge cases have been missed.
• This negotiation happens back-and-forth via files on disk until they agree on a testable “contract.”

The Evaluator then grades the final output against this mutually-agreed-upon contract, not the original high-level spec from the Planner. This process converts abstract user stories into tangible, testable assertions, ensuring the agent builds something that verifiably works.

Grading the Ungradable: Using Rubrics for Taste and Quality

For subjective qualities like UI/UX design, you can grade an agent’s output by establishing a strong, opinionated rubric. This involves defining clear criteria and weighting them. For example, a rubric for front-end design could include:

• Design: Overall aesthetic appeal.
• Originality: Avoidance of generic “AI slop” aesthetics like purple gradients.
• Craft: Attention to detail, polish.
• Functionality: Does it work?

This rubric is then calibrated by providing the Evaluator with few-shot examples of good and bad reference sites, aligning its “taste” with the developer’s own. In practice, this prevents agents from producing applications that look good on the surface but are functionally hollow or aesthetically generic.

The Art of Debugging: Read the Traces

Out of the box, models are often poor QA agents due to their inherent bias towards being agreeable. Making them effective requires significant prompt tuning. The primary debugging loop for improving the harness is not running more experiments, but reading the agent’s traces.

This process is akin to reading a stack trace. By meticulously reviewing the agent’s step-by-step reasoning, you can identify precisely where its judgment diverged from a human’s and then tune the system prompts or rubrics to correct that behavior. This is the most effective way to understand a model’s “spiky behaviors” and build a harness that compensates for its weaknesses.

As models improve, the harness must also adapt. Crutches for older models, like frequent context resets, may become unnecessary. The lesson is that the frontier of agent development doesn’t shrink; it just moves, and the harness must evolve with it.

• Tags:
• AI Agents
• Autonomous Agents
• Software Engineering
• Prompt Engineering
• Evaluation
• LLMs
• Feature
• Claude