Don't Trust Your AI Agents. Test Them Like a Scientist — Stephanie Jarmak, Omni Analytics
Stephanie Jarmak, who moved from astronomy into building coding benchmarks and agent evals, argues that most agent evaluations are easy to misread. She covers how harness choice can swamp model differences, why public benchmarks serve shortlisting rather than decisions, how to design experiment-style evals with realistic tasks, repeated runs, and full traces, and how to set quality gates, manage human review load, verify agent records, and treat production as the final test.
Agent results are easy to misread
Most agent evaluations look more conclusive than they are. In one early run at Sourcegraph, Stephanie Jarmak was testing whether a coding agent would use a code search MCP tool. She changed the initial prompt to push the agent toward the tool, and the run came back 28% cheaper. That was interesting, but it didn’t change any decisions, because the agent still never called the tool. The cost difference was an avenue to investigate, not evidence about the tool itself. A metric can move for reasons unrelated to what you are testing, and a headline number without verified behavior is noise.
The harness can matter more than the model
A third-party comparison on Terminal-Bench, using roughly 63 tasks, ranked models differently depending on the harness. With OpenHands, Opus came out ahead. When only the harness was swapped, the other model performed substantially better, a gap of more than 30%. Model and harness interact, so the configuration you evaluate has to be the one you will actually deploy, not the closest public approximation.
Public benchmarks shortlist, your own evals decide
An evaluation is measurement made to inform a specific decision about your product or process. Public benchmarks are better suited to surveying the landscape and building a shortlist. Model training makes results spiky across domains, so over-indexing on them is risky. A law firm, for example, shouldn’t steer decisions by terminal coding results unless it is bringing coding agents to its developers. Benchmarks are best treated as standardized reference sets. Teams can build internal benchmarks from real workloads and run an iterative flywheel, hill-climbing toward configurations that fit their environment.
The motivation behind CodeScaleBench was that SWE-bench was largely about bug fixing, while developers work across the whole software development lifecycle. A model can be strong at code understanding and weak at fixing bugs, so workload coverage matters. Tasks should be clustered into families, and tooling integration may need to differ per workload.
Designing evals like an experiment
Treat evals as experiments with several deliberate choices:
- Workload coverage: select tasks that reflect real work and group them into families.
- Repeated runs: agents are nondeterministic, so run-to-run variance has to be measured rather than assumed away.
- Interaction effects: changes to the model, harness, prompt, or tools can combine in non-obvious ways, and their downstream effects need to be observed.
- Observability: retain every trace, including failed attempts and non-use.
Trace the whole pipeline
Debugging the MCP case meant walking the pipeline step by step. Was the tool even in the agent’s context? If not, stop and fix that first. Was it called? What values came back? Was the returned information actually used? Improving a tool can still leave the end-of-task outcome unchanged, because the agent doesn’t leverage what it receives. The job is to act as a detective, following the chain from availability to call to use to outcome.
Choose what you actually measure
Metric choice encodes values. In a comparison of search tools, the goal was an F1 score that balanced precision and recall. A grep-based approach returned 295 files when only six were needed. If recall were the only concern, that breadth might be acceptable, but for this use case precision mattered. When reading single-number leaderboard scores, the question to ask is what the validator measured and why the authors considered it important. The right metric depends entirely on business needs.
Quality gates that actually stop work
Evals overlap with governance. Stephanie runs a software factory with approval queues, aiming to automate as much as possible while trusting these systems very little. In one case, the agent recognized that things were going wrong, but nothing in the system enforced a stop. Flagging and enforcement must be decoupled. An agent noticing a problem is a signal, not a gate. The gate has to live in the code itself, with a defined path for escalating to a human.
The human review bottleneck
A 1% false alarm rate sounds acceptable until every alarm is routed to a person. In one LLM-as-a-judge auditing setup, that rate would have meant about 10,000 reviews. Organizations need to ask whether they have domain experts with the capacity to review, and then tune alarm thresholds around that limit and around how consequential each alarm is for downstream effects.
When the agent’s record can’t be trusted
In one study, researchers seeded security issues into a task the agent was working on. A hash check on the environment passed and missed every injected issue. An independent record comparison covering traffic history and model changes caught all of them. An agent’s own trace is not automatically reliable. How much trust to place in the retained record should match the security needs and the measurement approach of the system.
Production is the final test
Realistic proxies are necessary, but production is the ground truth. At Omni Analytics, the team is classifying real user interactions, from easy Q&A to harder questions, dashboard building, and app building, to construct representative workloads. They are checking whether changes add value or simply shift failures downstream. After deployment, measurement has to continue in the real environment. Shadow deployments can help gather data where live changes carry a large blast radius.
Deciding when agents act alone
Risk tolerance determines when an agent may act autonomously and when it must stop and escalate. In Stephanie’s factory, agents handling lower-risk decisions state the next action, explain why human input is needed, define the operating scope, and suggest which human checks could be removed. Clear boundaries define when autonomous deployment is safe and when escalation is required.
Stephanie has published a GitHub repository of agent skills related to evaluation design, harness traceability, and reliability procedures, along with a lengthy internal best-practices document that agents can be pointed at to bring these practices into other systems.