Beyond the Benchmark: Measuring Quality for Web Research Agents

The speaker opens by framing the core problem: building a capable web research agent is only half the work, and the harder question is knowing whether it performs well in production. Coming from Parallel Web Systems, a startup that has grown to a reported $2 billion valuation, the speaker describes joining as one of the first ten engineers and spending roughly two and a half years building infrastructure, much of it from scratch. Building benchmarks for research agents became one of the company’s biggest challenges, and the talk walks through why the standard academic approaches failed for them and what they built instead.

What Parallel Builds

Parallel builds infrastructure for web search aimed at AI agents. The founding idea was to give agents the illusion of full, real-time access to the internet, so that an agent can answer questions about today’s weather, a customer’s recent activity, or a product launch, and know immediately when something on the web changes. The speaker points out that the brute-force approach of stuffing 100 billion web pages into an LLM context is impossible, since even large context windows hold on the order of a million tokens. The solution is a multi-layer pipeline that reimagines the search stack. The scale shrinks as you move up the stack: billions of pages at the crawl layer, thousands at ranking, tens of thousands at retrieval, and only a few hundred documents reaching the LLM. This determines how much computational intelligence can be spent at each layer, since cheap processing handles billions of documents while an LLM can only be afforded for a few hundred.

The products exposed to customers include a Search and Extract API (described as Google search for AI agents), a Monitor API (a push-based system akin to Google Alerts “on steroids” that wakes an agent when something relevant changes), and full research agents for teams that do not want to integrate tools themselves. The speaker claims their offering is Pareto-optimal on price-performance for web research agents, and attributes that result to their custom evaluation stack.

Mental Model of a Web Research Agent

The speaker offers a simplified model of the system under evaluation. An LLM sits on top of several information sources: vertical indexes (law, finance, B2B intelligence), a generic web index resembling Google, semi-public data providers such as PitchBook or Traction that require paid access, and general browser capabilities for taking actions like filling forms. The agent does more than call tools; it manages context, compresses and compacts information, and orchestrates everything, each of which is a point where things can go wrong. The evaluation system therefore has to capture failures in the LLM’s context handling and orchestration as well as in the individual tools.

Quality, Cost, and Latency

The speaker identifies three dimensions everyone cares about: quality, cost, and latency. Quality is considered far more important than the other two, because a bad agent is unusable regardless of speed or price. Cost and latency are comparatively easy to measure, since latency is a timer and cost is a token count. Quality is the hardest, because there is usually no ground truth to compare against.

Why Web Research Differs from Coding and Math

Coding and math are highly verifiable, since test cases can confirm correctness. Web research is not. Without a reference answer or ground truth, it is extremely difficult to know whether an agent is doing well. The field has responded with a mix of approaches: create some ground truth, then use an LLM as a judge to compare outputs against it. The speaker groups the resulting benchmarks into three buckets: short objective answers, list answers where precision and recall can be measured, and subjective long-form reports.

Existing Academic Approaches and Their Limits

The speaker walks through three benchmarks representing these buckets.

The first is a single-answer benchmark built by OpenAI, in which human annotators took documents from the internet and wrote questions about them. To filter out trivial questions, anything a person could solve in about ten minutes was discarded, leaving questions so hard that some could not be solved by a person even with four to six hours and full tools. Agents can now answer many of these, but the speaker’s realization is that these questions are not what anyone actually asks an agent. They are useful for benchmarking but amount to a marketing gimmick. The example given, asking about the fourth author of a paper meeting several narrow criteria, illustrates the point.

The second is Google’s list-style benchmark, which asks questions such as which US states have a certain fatality rate. The agent must research fifty states, find relevant statistics, filter, and return a list. This is more realistic and closer to production questions, though it still covers only one question type.

The third is a Chinese research group’s Deep Research Bench, which handles qualitative report-style questions, such as identifying the top five companies to invest in, where no single answer is strictly right or wrong. It scores reports along dimensions like readability, insight, comprehensiveness, and instruction following, using an LLM judge against a reference report. The speaker notes a critical gap: a beautifully written but hallucinated report would score well on these dimensions. The benchmark therefore adds a second metric, factuality, which checks the cited excerpts against the claims in the report.

The speaker’s central criticism of all these approaches is that they depend on hand-annotated questions, answers, and judges. That is acceptable for a one-off project, but it breaks down for a production system where customers keep arriving, agents and LLMs keep improving, and the questions customers ask keep changing.

The Evolution of Parallel’s Evaluation System

The company’s own path took about two and a half years and went through several stages.

First, they tried human labels, which failed. The agent was performing at around 90 percent, and roughly half of the 10 percent it was getting wrong was actually correct; the annotators were mistaken. Second, they replaced human annotators with synthetic ground truths and questions, borrowing from the BrowseComp approach but substituting an LLM for the human in the loop. This produced benchmarks that they hill-climbed on, but customers turned out to be asking entirely different questions. Third, they added sampling from production traffic. The speaker manually ran SQL queries against the database, found questions likely to be handled poorly, and ran them through the synthetic generation pipeline, roughly once a month. This was tedious and did not scale as customers were added.

The fourth and final step was to automate everything into a data flywheel of evals. Questions are sampled from production, answered by the agentic system, scored against rubrics captured during sampling, and evaluated on a recurring schedule. This addresses the problems of new customers signing up and of freshness, which the static academic benchmarks could not capture. The speaker also says the evals now cover four things that were missing from the academic benchmarks, though these are discussed later in the talk.

Sampling for Diversity and Hardness

There are two main goals in sampling: diversity and hardness. For hardness, the speaker describes a heuristic based on the cost of running the agent. Runs that are expensive, which often indicates an agent looping, asking repeated questions, or struggling after tool failures, are more likely to reveal problems. Correct runs tend to be cheap, while incorrect runs tend to be expensive, so sampling the costliest runs is an effective way to find hard questions. Other signals include strange outputs such as placeholders saying information is unavailable, empty outputs that may indicate compacted or lost information, nulls, and the confidence the system reports in its own answers. For diversity, the simple method is to sample across customers and ensure no single customer dominates the eval set.

Producing Reference Answers Without Human Labels

Because production questions come without ground truth, the team uses two techniques. The first is privileged labeling with an “Oracle” agent: an expensive system that is not subject to production budget or latency constraints and can use additional tools such as screenshots and image analysis. The second is a BrowseComp-style method that generates questions from sources actually used in production, rather than sources chosen at random. Once reference answers exist, the speaker argues that the judge matters a great deal. The team now uses an agent as the judge, not a bare LLM, and that agent has access to the same indexes and infrastructure, along with the reference answer and the output being evaluated.

Tracing, Masking, and Repeats

The speaker emphasizes that many people get the evaluation step wrong. Before any evaluation, everything must be traced. Tools such as Weave, which the speaker refers to as “Viv,” can produce a stack trace of an agent’s process, making it much easier to debug failures and to measure cost and latency.

The team also evaluates each component by masking parts of the system and running repeated evaluations. Repeats guard against LLM noise overwhelming the signal. Masking guards against a more subtle problem: a robust LLM can hide bugs by compensating for a broken tool. For example, the agent might fall back to a browser instead of the search tool, reach the correct answer, and do so at much greater expense.

The speaker gives a concrete case from a few months earlier. B2B customers wanted to research other businesses, including what they sell, what audits they hold, and whether they are compliant. The team built a benchmark for this company-research use case, and the pipeline scored 70 percent. That was puzzling, since they had a strong company index with tens of millions of companies and special embeddings. Masking the components clarified the situation. The pure web index alone reached 67 percent, while the agent using the company index reached only about 50 percent. The cause was that the agent struggled to call the company index tool, likely because post-training had favored the web over company data. After making the tool more agent-friendly, accuracy rose to 84 percent, and overall net accuracy reached 90 percent. The speaker stresses that without masking, the failure would have been nearly impossible to diagnose. The surface explanation looked like missing information in the company index, when the information was present and the agent simply failed to call the right tool. The 90 percent headline number is possible only when every component is inspected and brought to peak performance.

Reported Metrics

The evaluation reports correctness, comprehensiveness, grounding (factuality, the metric from the Chinese lab), cost, latency, and run-to-run variance across repeats. Together these show where agents perform well and where they do not, and they provide a path to improving the customer experience.

Open Challenges

The speaker acknowledges that the system, while largely automated, still has gaps, and the team spends most of its time on them.

The first challenge is that evaluating the evaluation system itself is very difficult. When a change is made to sampling, for example, it is unclear whether it is an improvement. The team has to look at downstream metrics and debug manually. Agents are used throughout, but a human working with an agent is still needed to determine whether changes have broken something.

The second challenge is subjective questions, such as “should I hire PR?” or any rating on a 1-to-10 scale where a score of six could be too low or too high depending on who is asking. There is no objective criterion, and the answer is almost personalized. The team has received complaints from both directions.

The third challenge is that the automated system depends on the company’s own infrastructure, which can itself be flawed. If a page that should have been indexed was not, the evaluation cannot detect it. The team currently works around this by collecting other metrics and measuring the gap separately.

The speaker closes by describing these as ongoing problems that the team is actively working to automate further, in order to keep pace with rapidly improving LLMs.

• Tags:
• Benchmarks
• Data Flywheel
• LLM as a Judge
• Tracing
• Agent Evaluation
• Web Research Agents
• Synthetic Ground Truth
• Production Sampling
• Component Masking
• BrowseComp
• Deep Research Bench
• Factuality Grounding