Nathanael Weill of Air Canada told AI Engineer he builds tests by planting the answer first, then asking the agent to find it.
He showed the pattern first on a 2024 RAG assistant for telecom engineers that had to answer from about 10,000 vendor documents, where finding the right page meant getting the right answer 90 percent of the time, according to Weill.
Weekly expert panels could grade only one variant and often disagreed, so Weill reversed the loop and had an LLM write questions from a known page and graded retrieval with code.
At Air Canada the same bottleneck returned in 2026 on supplier contract review, where a four-step agent must spot clauses, retrieve supporting documents, draft amendments and classify them, and no public benchmark covers that workflow, Weill said.
He detailed five code-checkable tricks his team now uses instead of LLM as a judge, including planting a rule like a $1 million cap in a generated contract, deleting a required clause from a signed template, and permuting invariant fields such as supplier names, which together yielded 275 deterministic tests and cut feedback from four weeks to five minutes, according to the talk.
That design sits apart from Google Developers Blog guidance to anchor tests in critical user journeys rather than abstract benchmarks, and from DeepEval guidance that production evals must run asynchronously without blocking responses.
Weill warned that LLM as a judge added more noise than the agent itself and said his team reengineered parts of the agent to make them evaluable with exact match or code, a tradeoff that speeds iteration but narrows what gets measured.
He did not say how those synthetic contracts map to the legal risk of a missed clause or whether coverage of unknown rules can ever be proven without the very experts the method was built to save time from.
