RL Environments Explained: How AI Agents Learn Real-World Work | Brendan Foody, Mercor

Brendan Foody, CEO of Mercor, introduces RL environments as the new frontier in AI training data, marking a significant shift from the crowdsourcing era of behavior cloning to an “agentic era” focused on high-skilled experts. This evolution, observed as early as 2024, prioritizes sourcing top experts—including software engineers, lawyers, doctors, and bankers—to build “frontier evals” and RL environments. Mercor emerged as a leading vendor for both frontier labs and application-layer companies, emphasizing RLVR (Reinforcement Learning from Virtual Reality), which integrates rich applications and worlds to teach agents tool use.

Components of an RL Environment

An RL environment comprises three core elements:

• Worlds: These are comprehensive datasets, including messages, slides, documents, and spreadsheets, simulating real-world project or company contexts.
• Apps: High-fidelity clones of widely used applications like Salesforce, ServiceNow, or Microsoft 365. Agents interact with these via mechanisms like MCP, CLI, or Kua.
• Tasks: Specific prompts coupled with “verifiers,” which can be rubrics or unit tests, used for both evaluation and training.

The primary challenge lies in scaling these environments to cover the vast distribution of worlds, apps, and tasks across the entire economy, a process heavily reliant on human expertise, albeit with models in the loop. Mercor has witnessed explosive growth in expert engagement, processing 2.5 million expert hours in Q2 alone, highlighting the scale required.

The Indispensable Role of Humans

Foody stresses that only humans can reliably measure the frontier in most domains. While exceptions like mathematics allow for clean simulation, tasks such as building a slide deck reveal models’ difficulty in self-identifying errors. This necessitates human-designed rubrics, akin to a professor grading an essay, to provide crucial feedback. Building these verifiers is inherently complex, requiring a deep understanding of the problem space, including potential solution paths and myriad mistakes.

A Legal RL Environment Walkthrough

Foody presents a real legal RL environment developed in collaboration with top law firms. Lawyers craft real-world project scenarios and detail data room outlines, which are then populated with model assistance. These data rooms are rendered into cloned apps (e.g., Google Workspace), and agents are prompted to perform tasks like evaluating legal liabilities. Crucially, human lawyers create rigorous rubric criteria to ensure accurate model responses and prevent “reward hacking,” a technically challenging process involving extensive research and quality control. This process generates high-quality verifiers and leaderboards, which now feature advanced models like GLM 52 and Chimera K3, democratizing access to frontier intelligence.

Post-Training Results and Data as a Moat

Demonstrating the impact, Foody shares post-training results on Apex Agents, a legal dataset. A GLM 47 model, after training on 1,800 tasks with approximately $500k in compute, showed dramatic improvements: corporate law performance jumped from 4.7% to 26.6%. Significantly, these gains generalized to other benchmarks, including GDP eval and Apex V1, underscoring the value of such domain-specific data. Foody asserts that datasets are becoming the “moat” for application-layer companies, enabling them to “own their own intelligence.”

Mercor’s Data Acquisition Models

Mercor offers three primary ways for companies to acquire data:

1. By Task: The most common, involving custom per-task pricing for complex, specific environments. Frontier labs can purchase tens of thousands of tasks monthly.
2. Off-the-shelf Data: Mercor provides pre-built, high-investment datasets, particularly appealing to new labs seeking efficiency by avoiding redundant data creation.
3. Hourly Model for Experts: Less central now, where Mercor supplies experts and customers manage their organization.

Q&A Highlights

Data Pricing and Quality: Data pricing is tied to desired model improvement and Mercor’s cost structure, ranging from $50 to $10,000 per task. Data quality encompasses realism (reflecting real-world distributions) and verifier accuracy. Mercor employs trajectory analysis and human review to ensure rubrics align with human judgment.

Synthetic Data: Foody clarifies that RLVR itself leverages synthetic data by generating and scoring model trajectories. While models aid in environment population, humans remain vital for measuring capabilities “beyond the frontier”, as models cannot reliably self-evaluate at the bleeding edge.

Future of RL Environments: The current focus on tool use is expanding. Future directions include ultra-long-horizon tasks (agents performing tasks requiring 100s or 1000s of human hours) and integrating virtual co-workers to address the significant gap in measuring agents’ social interaction capabilities.

Scaling Rubric Generation: While AI copilots can enhance expert efficiency, direct model generation of robust rubric criteria remains challenging due to noise. Task creation, particularly for ensuring grounding in reality, largely still requires human expertise. An exception is cyber defense, where attacker/defender agents can sometimes replace human verifiers for certain provable domains.

Base Model Limitations and Partnership: The effectiveness of training depends on the base model’s trainability, indicated by its ability to achieve some correct trajectories within a set (e.g., “pass at 16”). Mercor advises companies to partner for custom data needs, benefiting from its specialized infrastructure and talent network, which offer significant economies of scale over in-house solutions.

• Tags:
• Synthetic Data
• Large Language Models
• Reinforcement Learning
• Feature
• Human Feedback
• Tool Use
• Data Quality
• Mercor
• LLM Fine-Tuning
• Frontier AI
• RL Environments
• Agentic Data
• Data Pricing
• AI Training Data