# Don't trust agents. Test them like scientists.

_Stephanie Jarmak told AI Engineer teams must test AI agents like scientists after a 28% cheaper run still failed to use its tool._

**Published:** 2026-10-23
**Source:** https://www.startuphub.ai/ai-news/technology/2026/don-t-trust-agents-test-them-like-scientists

---

Stephanie Jarmak told [AI Engineer](https://www.youtube.com/watch?v=A3FL_zTQKU0) to stop trusting AI agents and test them like scientists, citing a 28% cost drop that masked a complete tool-use failure.

She showed 63 Turbo Bench 4 tasks where swapping only the harness flipped results by more than 30% between [Opus 5](https://www.startuphub.ai/ai-news/artificial-intelligence/2026/anthropic-launches-claude-opus-5) and Astra.

Jarmak, an AI engineer at Omni Analytics and former Sourcegraph evaluation lead who built Code Scale Bench, said teams must "relentlessly question everything" and keep full traces to diagnose why agents ignore tools.

She measured [Claude](https://www.startuphub.ai/ai-news/artificial-intelligence/2026/anthropic-formalizes-ai-services-with-new-partner-tiers) returning 295 files when 6 were needed with grep, a case where optimizing for recall hides poor precision and inflates F1.

She warned traces themselves can lie, noting a hash check passed despite injected security issues until an independent record comparison detected every change.

Her software factory relies on approval queues and audits, but a 1% false alarm rate would still force 10,000 human reviews with no clear capacity model.

Public coding benchmarks like SWE-Bench Pro now average 107.4 lines across 4.1 files per Scale, yet Jarmak said production is truth and teams must measure in shadow before granting autonomy, pointing to a GitHub repo and 300-page archive teams can use now.

---

Original analysis from [startuphub.ai](https://www.startuphub.ai), the #1 AI startup directory.

© StartupHub.ai. All rights reserved. You may not republish this article in full without a license. Search engines and AI research tools may crawl and summarize for reference. Bulk reproduction or model training on this content requires a license. See https://www.startuphub.ai/terms.