From Agent Traces to Agent Simulations — Rustem Feyzkhanov, Snorkel AI
Rustem Feyzkhanov discusses the critical need for companies to build private, production-aligned benchmarks for AI agents. He explains how to turn agent traces into repeatable simulations, why public benchmarks are insufficient, and how a CI pipeline for agents, integrating observability and experimentation, can ensure reliable evaluation, continuous improvement, and effective release management, moving beyond simple pass rates to measure cost, latency, and policy adherence.
Rustem Feyzkhanov’s talk underscores the critical need for every company to develop its own private benchmarks for AI agents. He argues that this is the only reliable way to evaluate, release, and continuously improve agents, emphasizing that these benchmarks must closely mimic real production environments, including tools, API services, policies, and workflows. Crucially, they should be a dynamic part of the agent’s lifecycle, constantly updated with data from production traces.
Limitations of Production Traces and Public Benchmarks
Production traces, while useful for identifying failures, lack repeatability due to variations in database states and tool versions, preventing true “apples-to-apples” comparisons. Offline simulations address this by converting production traces into repeatable experiments, enabling comprehensive evaluation across metrics like cost, latency, retries, and success rate.
Public benchmarks (e.g., SweepBench, TerminalBench, Co-Bench) are domain-specific and often prioritize pass rates. While they help in