How to Test AI Agents When No Benchmark Exists — Nathanael Weill, Air Canada
Nathanael Weill of Air Canada explains how to evaluate AI agents when no public benchmark exists and expert time is scarce: start from a known answer, generate the input from it, and check whether the agent recovers it. He covers lessons from a telecom RAG project that moved from one expert-graded test a week to dozens a day, and five ways to build test sets for a supplier contract review agent, along with why code-based checks are preferred over LLM judges.
