Stephanie Jarmak told AI Engineer to stop trusting AI agents and test them like scientists, citing a 28% cost drop that masked a complete tool-use failure.
She showed 63 Turbo Bench 4 tasks where swapping only the harness flipped results by more than 30% between Opus 5 and Astra.
Jarmak, an AI engineer at Omni Analytics and former Sourcegraph evaluation lead who built Code Scale Bench, said teams must "relentlessly question everything" and keep full traces to diagnose why agents ignore tools.
She measured Claude returning 295 files when 6 were needed with grep, a case where optimizing for recall hides poor precision and inflates F1.
She warned traces themselves can lie, noting a hash check passed despite injected security issues until an independent record comparison detected every change.
Her software factory relies on approval queues and audits, but a 1% false alarm rate would still force 10,000 human reviews with no clear capacity model.
Public coding benchmarks like SWE-Bench Pro now average 107.4 lines across 4.1 files per Scale, yet Jarmak said production is truth and teams must measure in shadow before granting autonomy, pointing to a GitHub repo and 300-page archive teams can use now.
