"Intelligence is measured by the efficiency of skill acquisition on unknown tasks." This foundational insight, articulated by François Chollet, creator of Keras and the Abstract and Reasoning Corpus for Artificial General Intelligence (ARC-AGI), underpins a critical shift in how the AI community evaluates progress. In a recent interview at NeurIPS 2025, Y Combinator General Partner Diana Hu sat down with Greg Kamradt, President of the ARC Prize Foundation, to dissect why many prevailing AI benchmarks fall short and how ARC-AGI is redefining the pursuit of human-like generalization. Their discussion highlighted a crucial distinction between mere performance and genuine intelligence, a topic of paramount importance for founders, VCs, and AI professionals navigating the rapidly evolving landscape.
The existing paradigm of AI evaluation, often focused on benchmarks like MMLU, has inadvertently steered development towards models that excel at memorization and brute-force computation rather than true understanding or adaptability. As Kamradt noted, "You would normally think that intelligence would be how much can you score on the SAT test, or how hard of math problems can you do." While impressive, achievements in areas like chess, Go, or self-driving demonstrate superhuman skill in specific, often pre-defined domains, but not necessarily the fluid intelligence required to rapidly acquire new, unrelated skills. This narrow focus creates a deceptive sense of progress, akin to "PhD++ problems" that simply demand more data or compute, rather than novel reasoning.
Chollet’s definition posits that true intelligence lies in the efficiency with which a system can learn new things. This is the core tenet of ARC-AGI, which was initially introduced in 2019. Its tasks are deliberately designed to be trivial for humans but profoundly challenging for machines relying solely on pre-training or massive datasets. Early large language models, prior to 2024, performed dismally on ARC-AGI 1.0, achieving only 4-5% accuracy. This stark contrast underscored the benchmark's ability to expose fundamental limitations in generalization, revealing that current AI, despite its apparent prowess, often lacked the human-like capacity to infer underlying rules from minimal examples and apply them to novel situations.
However, a significant shift occurred with the advent of advanced reasoning paradigms. Kamradt recounted that when O1 models (presumably referring to a recent major model release) first emerged, performance on ARC-AGI jumped to 21%. "That tells you something really interesting is going on," he stated. This leap indicated that AI was beginning to exhibit nascent reasoning capabilities, moving beyond sheer memorization. The community, including major labs like OpenAI, xAI (with Grok-4), Google DeepMind (Gemini 3 Pro), and Anthropic (Opus 4.5), has since recognized ARC-AGI’s utility, incorporating it into their model release evaluations.
This widespread adoption, while validating ARC-AGI's relevance, also brings a cautionary note regarding "vanity metrics." Kamradt emphasized that while the community's recognition is positive, it does not signify the completion of ARC Prize's mission. The ultimate goal is "to pull forward open AGI progress" by inspiring researchers to develop genuinely generalizable systems. Simply optimizing for a benchmark, even one as thoughtfully designed as ARC-AGI, risks falling into the trap of over-fitting without achieving true underlying intelligence. The aim is not just to pass the test, but to understand why and how the system passes it, ensuring that the progress is transferable and not just a domain-specific triumph.
