If You Score 100%, Your Test Failed

Ramakrishnan Lokanathan told AI Engineer that if your eval hits 100%, your eval is wrong, and he demoed a way to prove it.

An eval, an abbreviation of evaluation, is a task with standards and evidence to judge how a system will perform when it matters, and Lokanathan said most teams overcomplicate it or index on one signal. He compared it to exams like the SAT, which he called one signal of college readiness, not a full measure of ability, and said the same holds for SWE-Bench or the humanities law exam for models. A single score never captures the job.

The fix starts with ambition, he argued. Define the most ambitious success first, assume AGI, and ask what must happen, what must never happen, and when the agent should stop. Then test the whole workflow, not just the final answer. He showed a simple image edit where the prompt was only change the background, the model did change it to cream but also added a second handle, and a check that only looked at start and end would have passed it.

That points to a holistic evaluation system, Lokanathan said, with three layers. Automated checks run the eval questions in CI on every deploy, humans review the results instead of letting an agent grade itself, and adversarial tests try to break the workflow on purpose. His recent work for the third layer is E-Slams, electronic slams, an open-source Python package on PyPI that makes any agent play games to reveal how it reasons. It supports about 50 games and 90 providers, per his talk, from chess and Go for planning with hidden state to poker for decisions under hidden information to cooperation and arcade games, and the name itself is noisy, Merriam-Webster lists 342 synonyms for slams. You install the package, pit your old agent against a new model, and watch the sequence, not just the winner. In his preview on eslams.com, Fable 5.1 played tic-tac-toe against GPT-6 Astra and tied, a result he said tells you more about recovery from blunders and shifting strategy than a single correct answer on SWE-Bench.

Then evolve the tests, or they rot. Lokanathan cited the MIT GenAI Divide finding that 95% of organizations see no return because the product does not deliver its promise in production, and he pointed to Google Assistant moving to Gemini as a regression his framework would have caught, when reminders that worked before suddenly drew the reply that a language model cannot set reminders. Keep past wins and past failures in the suite, add fresh challenges when scores creep to 85 or 90, and turn every real-world failure into a future test through observability and simple thumbs up or down feedback. If your suite is easy, make it harder. That is the signal to change the test, not to celebrate.

Lokanathan, who built on OpenAI GPT-3 in 2022, later sold a pet-care assistant and now red teams and researches at Lawrence Berkeley Lab and with the London Business School Data Science and AI initiative, said evals may matter less in ten years if capabilities keep rising, but today they decide whether a high-stakes agent keeps its promise. E-Slams will host what he calls the Grand Slams of AI in February, an open contest for models to play. You can try it now against your own agent, and watch whether it handles long chains of decisions without letting an early move become a liability later.