Rayan Garg on Why Long Horizon AI Agents Need Better Verifiers

As autonomous AI agents attempt to handle tasks that stretch over hours or days, measuring their actual performance has become one of the hardest challenges in machine learning. In a technical talk, Rayan Garg from Theta Software broke down why existing benchmarks for long horizon work fall short and how the industry must rethink environment design and task verification.

Who Is Rayan Garg

Rayan Garg is a researcher and engineer at Theta Software, where his focus centers on creating dependable evaluations, environments, and verifiers for autonomous agents. His work addresses the growing gap between headline benchmark metrics and real-world agent performance.

The Flaws in Current Long Horizon Metrics

Industry benchmarks often measure long horizon performance using a simple time threshold. For instance, an evaluation might define a model as capable of long horizon tasks if it successfully completes work at the 16-hour mark. Garg argues that this approach produces noisy, unreliable data.

Wall clock time hides the true difficulty of a task. Human time estimates vary wildly depending on skill level and experience. Furthermore, an artificially stretched task with serial dependencies might take hours to complete without testing complex reasoning. Real difficulty stems from state complexity and high-stakes choices, not just elapsed time.

Cascading Errors and State Space Complexity

True long horizon difficulty occurs when a single mistake early in a process cascades through every subsequent step. A bad initial database query or misconfigured environment setting corrupts all downstream actions, turning a simple task into an unrecoverable failure mode.

Evaluating models under these conditions requires creating realistic, dynamic environments. Standardized evaluation becomes exceptionally difficult as state spaces grow. When state spaces balloon, traditional judge models often struggle to accurately evaluate whether an agent truly succeeded.

Verifying Success from Final State

To avoid relying on subjective judge estimates, Garg highlights the importance of verifying agent success from the actual final state of the environment. Rather than grading step-by-step trace files or trusting a judge model to guess correctness, system designers should evaluate concrete final artifacts.

This approach involves collapsing massive state spaces using sample trajectories and structured tools. Evaluators must also ensure LLM judges do not see information they should not, preventing data leakage and unearned success scores.

Reusing Agents for Automated QA

Another major strategy Garg shares is using specialized agents to sift through artifacts like CI logs, build outputs, and execution traces. By pairing final-state verification with specialized agentic grading, teams can build clear rubrics and quality assurance pipelines.

Ultimately, progress in long horizon autonomous work lives or dies on environment and verifier design. Simply pushing headline benchmark scores without honest evaluation frameworks will fail to produce reliable agents in production.