How to Test AI Agents When No Benchmark Exists — Nathanael Weill, Air Canada
Nathanael Weill of Air Canada explains how to evaluate AI agents when no public benchmark exists and expert time is scarce: start from a known answer, generate the input from it, and check whether the agent recovers it. He covers lessons from a telecom RAG project that moved from one expert-graded test a week to dozens a day, and five ways to build test sets for a supplier contract review agent, along with why code-based checks are preferred over LLM judges.
Writing the answer first
Nathanael Weill, who leads a team building AI agents at Air Canada, starts from a problem many applied teams face: the tasks are so specific that no public benchmark exists, and the domain experts who could judge the output are scarce and busy. Without a way to grade results, every change to an agent is a guess. His answer is to invert the usual order. Instead of asking a question and then waiting for an expert to check the response, you start from a known answer, generate an input from it, and check whether the agent can find its way back to that answer.
Lessons from a telecom RAG project
The approach grew out of a 2024 question-answering assistant for telecom engineers, built over a corpus of roughly 10,000 documents plus a large body of vendor material. The team had limited access to experts and could not reliably grade answers themselves. Their system competed with other teams’ solutions, and it succeeded in the practical sense that people adopted it, though Weill admits they never fully understood why it outperformed the alternatives.
The team observed a pattern familiar to anyone building RAG systems: once retrieval surfaced the right page, the answer was usually correct, so the real challenge was finding the right documents. They built an increasingly complex pipeline, but published techniques that promised gains did not transfer to their setting, and each new layer tended to break behavior that had previously worked. Complexity did not reliably help. Changes could be made in minutes, yet grading them required a weekly meeting in which experts compared five chatbot answers side by side, often disagreeing with one another. The budget was one meaningful trial per week, which was far too slow to iterate.
The fix was to reverse the problem. Start from a page, ask an LLM to write a question whose answer lives on that page (an easy generation task), run the agent on the question, and compare its retrieved sources and answer against the source page. Because the reference is known, grading becomes a code check rather than an opinion. This moved the team from one evaluation a week to dozens a day. With fast feedback, they found that removing complexity helped, and that choices such as the embedding model, chunking strategy, and BM25 weighting mattered more. Accuracy rose week over week. The lesson he carried forward: if you cannot grade the agent, you cannot make it better.
Applying the idea to supplier contract review
In 2026, Air Canada’s data science team was asked to automate supplier contract analysis. Experts read contracts clause by clause, flag non-compliant terms, and propose rewrites. Missing a clause is expensive because it can backfire on the company later, and slow review delays supplier onboarding. The agent has four stages: classify contract sections into domains and detect defects or needed changes, retrieve the documents that support an amendment, draft the amendment, and then classify whether that amendment should be kept. Retrieval was already covered by the RAG lessons. The other stages had no established evaluation method, so the team had to design test sets for them.
Five ways to build a test set
- Past decisions: Pair an original signed contract with its later revision, diff the two as you would code, and expect the agent to produce the same flags. This is the most intuitive approach, but the team usually kept only the latest version, with no record of the negotiation history, so the pairs largely did not exist.
- Rules as the seed: Start from a known rule, such as a $1 million threshold that triggers specific handling. Write a clause that respects the rule, generate the surrounding contract, then break the rule and expect the agent to flag it. Experts and existing documentation supplied some rules, the team added its own, and together they produced 275 tests with known answers. Coverage is only as complete as the rule inventory.
- Removed clauses: Take a signed contract, delete a clause, and expect the agent to report that something is missing. This is the hardest and most revealing test, because supplier templates differ and deciding what belongs in a given contract requires real judgment. Mixed with contracts that should produce no flags, it became the strongest benchmark, and the agent missed most of the removed clauses.
- Changes that shouldn’t matter: Swap the supplier’s name or address, reorder clauses, or rerun the same contract. The number and nature of flags should stay constant. These invariance checks are all verifiable in code.
- Templates with blanks: Experts already work from templates with placeholders. Rather than filling the blanks by hand, the team generated arrays of candidate values for each blank, producing large numbers of contracts, some of which break rules and some of which do not. The correct answer is known by construction. Building this pipeline took about four weeks to get feedback, but once in place, the evaluation script returned results in roughly five minutes.
Why code beats LLM judges
Weill ranks grading methods by reliability. Exact match and code checks are fast and deterministic. LLM-as-a-judge can be powerful, but on this project the noise it introduced was comparable to the noise of the agent under test. Finding the right prompt, harness, and verification scaffolding for the judge was nearly as hard as solving the original problem. Where possible, the team designs tests so that correctness can be verified in code, and they reserve judges for cases where nothing else works.
The same principle appears in other patterns. For agents that take multi-step actions, the expected trajectory can be defined in advance and the trace checked against it. For text-to-SQL-style tasks, query results can be compared directly. Not every system fits this paradigm, and in some cases the team re-engineered its solution toward a design that could be evaluated at all, even though it departed from the original architecture.
Expert feedback improves the tests as well as the agent
Expert feedback does not only serve to patch the agent. It becomes input for building better tests, including new prompts, new ways of removing clauses, and new edge cases. Each loop improves both the system and the benchmark used to measure it. Weill is clear about the limits of this process: a passing test suite is not a go/no-go decision for release. It is a tool for anticipating where the agent will fail before experts encounter those failures.