AI Agents Stall on Core AI Research

The promise of explosive AI progress often hinges on AI agents automating AI research itself. However, tangible evidence supporting this vision remains surprisingly scarce. Existing evaluation methods fall short. Some focus on narrow, verifiable tasks, inherently excluding the open-ended nature of true research. Others submit AI-generated papers to blind peer review, a process already strained, unpredictable, and prone to inconsistent quality.

Shadow Evaluations: A New Metric for AI R&D Automation

To address this measurement gap, researchers Peter Kirgis, Sayash Kapoor, Andrew Schwartz, Stephan Rabanser, David Africa, and colleagues introduced shadow evaluations. This novel approach places an AI agent at the heart of a high-quality, unpublished research paper, tasking it with tackling the central, open-ended research question. Crucially, the original authors of the paper then grade the agent's output. This method provides a direct, author-centric assessment of AI's research capabilities.

Frontier Agents Fall Short on Research Insight

In their study, the researchers applied shadow evaluations to two unpublished NeurIPS 2026 submissions. Frontier agents were given six days and substantial compute resources, costing thousands of dollars. While the agents successfully completed all engineering tasks without human intervention, they failed to make significant progress on the core research questions. The authors unambiguously rejected the outputs, highlighting five persistent failure modes: inadequate judgment of publishable research standards, uncreative responses to research design flaws, ineffective backtracking from dead ends, poor resource awareness, and instruction drift. A subsequent robustness check using a different model and scaffolding reproduced these failures, reinforcing the findings.

The Engineering-AI Divide in Research

The results offer a sobering early assessment: today's most advanced AI agents excel at the engineering components of AI research, coding, implementation, and execution. Yet, they demonstrably struggle with the more nuanced, critical aspects of the research lifecycle. This includes formulating novel hypotheses, designing robust experiments, interpreting ambiguous results, and critically evaluating the significance of findings. The current limitations suggest that while AI can automate the 'how' of research, the 'what' and 'why' remain firmly in the human domain for now. This distinction is vital for investors and founders considering the trajectory of AI-driven innovation.