ClawBench: Testing Real-World AI Agents

The automation of routine online tasks remains a significant hurdle for the widespread adoption of AI agents as general-purpose assistants. While AI can manage an inbox, the complexity of everyday digital interactions presents a formidable challenge.

Bridging the Gap: The ClawBench Framework

To address this, the researchers introduce ClawBench, a novel evaluation framework designed to test AI agents on 153 real-world tasks across 144 live platforms. Spanning categories from e-commerce to job applications, ClawBench demands capabilities beyond current benchmarks, including information extraction from user documents, multi-step cross-platform navigation, and extensive form completion. Crucially, ClawBench operates on production websites, mirroring the dynamic and complex nature of real-world web interactions, unlike static, offline sandboxes.

Performance Realities: Frontier Models Fall Short

Evaluations of leading AI models, both proprietary and open-source, reveal a stark reality: current frontier models can only successfully complete a small fraction of these everyday tasks. For instance, Claude Sonnet 4.6 achieved a mere 33.3% success rate. This highlights a substantial gap between current AI agent capabilities and the requirements for reliable, autonomous operation in the real world. The development of robust ClawBench AI evaluation benchmarks is essential for driving progress towards truly capable AI assistants.

The Path Forward: Real-World Evaluation for General Assistants

The ClawBench AI evaluation framework offers a critical step towards developing AI agents that can reliably handle diverse, everyday online tasks. By moving evaluations to live production environments, the framework provides a more accurate assessment of agent performance and identifies key areas for improvement. Progress on benchmarks like ClawBench is paramount for realizing the vision of AI agents as indispensable, general-purpose assistants.