Rethinking Environments for Long-Horizon Work — Rayan Garg, Theta Software

The presentation by Rayan Garg of Theta Software critically examines the concept of “long horizon” tasks for AI agents, arguing that its definition and measurement are far more nuanced and impactful than commonly perceived. Progress in autonomous AI hinges not just on headline benchmarks but fundamentally on well-designed environments and robust verifiers.

Defining “Long Horizon”

The “time horizon” at which AI agents can operate autonomously is rapidly accelerating. While benchmarks like Meter offer a popular approach by linking AI agent performance to human time estimates (e.g., a 16-hour threshold for a 50% success rate on tasks taking humans 16 hours), this metric is inherently scalar and evolves constantly. What was considered “long horizon” a year ago is no longer today.

The definition can be approached in two ways:

• Human-centric: Measures against human time taken, often using expert-derived thresholds.
• Model-centric: Uses internal model units like tokens consumed, steps taken, or tool calls. This provides insights into the intrinsic difficulty of a task for an AI agent, especially for tracking the technical frontier (e.g., observing a model handle millions of tokens due to increased context windows). However, these metrics are noisy and depend heavily on the specific model and harness used.

Both approaches have significant weaknesses. Human-centric metrics can be misleading because tasks tedious for humans might be trivial for an AI (e.g., complex Excel formatting), and the methodology for human time estimation is highly variable and can be inaccurate, especially for tasks pushing the limits of human capabilities. Conversely, model-centric metrics don’t directly translate to human understanding. Given the diverging paths of human and AI capabilities, a holistic view combining both perspectives is crucial.

Measuring Model Capabilities Beyond Task Length

Beyond simply task length, evaluating true model capabilities requires considering:

1. Environment Complexity:

   • Tool Coordination: The number of external dependencies and tools (e.g., Grafana, GitHub, AWS CloudWatch, databases) an agent must coordinate and move information across. Tasks become genuinely complex when agents need to interact with multiple, disparate systems.
   • State Changes: The degree to which the environment’s state evolves throughout the task. Artificially long tasks can be created by chaining independent sub-tasks (paralyzable complexity, like analyzing different files in a codebase in parallel). True long-horizon difficulty arises from sequential complexity, where early decisions or misreads cascade and have significant downstream consequences (e.g., a bad query in a log dashboard leading to incorrect subsequent actions).

2. Ambiguity: The initial information (instructions, artifacts) provided to the agent.

   • Human work often involves high ambiguity and exploration. Testing agents’ ability to explore and cope with incomplete information is vital.
   • However, increased ambiguity leads to a wider space of possible correct solutions, making standardized evaluation much harder.

The Critical Role of the Verifier (Judge Model)

As tasks grow in complexity and length, especially in “software-failing domains” where deterministic verification (like running unit tests) is impractical or impossible, the role of a verifier (often a “judge” or “critic” model) becomes paramount. The judge’s primary function is to determine correctness and assign a reward signal during training.

Judges typically evaluate two aspects:

• The final state of the environment after the agent’s actions.
• The trajectory (sequence of steps and decisions) the agent took.

Evaluating the trajectory is crucial for preventing “reward hacking,” where an agent might find an unintended shortcut or exploit vulnerabilities (e.g., escaping a sandbox) to achieve a desired final state without genuinely solving the problem. Overly rigid guidance or simply comparing against a single “reference solution” for ambiguous tasks is ineffective, as many correct paths might exist.

Designing Effective Judges and Rubrics

Effective judge design is as complex as agent design itself:

• Judges are Agents Too: Judges often require access to the same environmental tools and interfaces as the agent to verify complex outcomes (e.g., checking GitHub CI/CD logs or AWS CloudWatch after a deployment task). Safeguards, such as read-only permissions, are essential to prevent the judge from accidentally mutating the environment.
• Processing Long Trajectories: For very long and complex agent trajectories, simply feeding the entire sequence into an LLM’s context window is unfeasible. Instead, sophisticated processing is needed, such as storing trajectories in databases, using sub-agents for information enrichment, parsing trajectory phases (e.g., log analysis, code writing, verification), and making the trajectory queryable through metadata.
• Learnability and Reward Density: The quality of the “reward signal” provided by the judge’s rubric is vital for agent learning. While a dense reward signal is desirable, overly complex rubrics for frontier problems can lead to inconsistent judge application, necessitating rigorous QA (gold standards, no-op variance, coverage, expert agreement). Emerging patterns include using deterministic verifiers in conjunction with judges and dynamic evaluation rubrics that allow for partial credit.

Critiquing Current Benchmarks

The speakers point out significant flaws in many widely used benchmarks (e.g., GDP valer, Toolbench, Apex agents), particularly in finance:

• Not Truly Long Horizon: Many tasks within these benchmarks do not meet the human-hour thresholds established for “long horizon” work.
• Saturated and Easy: Many benchmarks are already “reasonably saturated,” meaning models achieve high success rates (e.g., pass@1 of 57% for Apex agents IB implies 100% solutions for a large portion of tasks), indicating they are not pushing the frontier of agent capabilities.
• Narrow Breadth: Benchmarks often focus on a limited set of tasks (e.g., Excel tasks in GDP valer, IB in Apex agents), neglecting crucial, harder domains within broader fields (e.g., credit, debt, risk in finance).
• Insufficient Reward Signal: Existing benchmarks often lack the granular and detailed rubrics necessary for effective model training, contrasting with Theta’s approach of employing 20 sub-criteria with 10 sub-criteria per main criterion.

Theta Software’s own finance data, designed with these principles, demonstrates that even for tasks averaging 15 human hours, models still struggle significantly (mean 5 on their metrics), highlighting the gap between current benchmarks and truly challenging long-horizon work. The central message is that robust environment and verifier design are paramount for actual progress in autonomous AI.

• Tags:
• AI Agents
• Feature
• LLM Evaluation
• Reward Hacking
• Long-Horizon Tasks
• Autonomous AI
• Evaluation Metrics
• Environment Design
• Verifiers
• Judge Models
• Benchmark Limitations
• Software-Failing Domains
• Tool Coordination
• State Changes
• Ambiguity in AI
• Rayan Garg