"Intelligence is measured by the efficiency of skill acquisition on unknown tasks." This foundational insight, articulated by François Chollet, creator of Keras and the Abstract and Reasoning Corpus for Artificial General Intelligence (ARC-AGI), underpins a critical shift in how the AI community evaluates progress. In a recent interview at NeurIPS 2025, Y Combinator General Partner Diana Hu sat down with Greg Kamradt, President of the ARC Prize Foundation, to dissect why many prevailing AI benchmarks fall short and how ARC-AGI is redefining the pursuit of human-like generalization. Their discussion highlighted a crucial distinction between mere performance and genuine intelligence, a topic of paramount importance for founders, VCs, and AI professionals navigating the rapidly evolving landscape.
The existing paradigm of AI evaluation, often focused on benchmarks like MMLU, has inadvertently steered development towards models that excel at memorization and brute-force computation rather than true understanding or adaptability. As Kamradt noted, "You would normally think that intelligence would be how much can you score on the SAT test, or how hard of math problems can you do." While impressive, achievements in areas like chess, Go, or self-driving demonstrate superhuman skill in specific, often pre-defined domains, but not necessarily the fluid intelligence required to rapidly acquire new, unrelated skills. This narrow focus creates a deceptive sense of progress, akin to "PhD++ problems" that simply demand more data or compute, rather than novel reasoning.
Chollet’s definition posits that true intelligence lies in the efficiency with which a system can learn new things. This is the core tenet of ARC-AGI, which was initially introduced in 2019. Its tasks are deliberately designed to be trivial for humans but profoundly challenging for machines relying solely on pre-training or massive datasets. Early large language models, prior to 2024, performed dismally on ARC-AGI 1.0, achieving only 4-5% accuracy. This stark contrast underscored the benchmark's ability to expose fundamental limitations in generalization, revealing that current AI, despite its apparent prowess, often lacked the human-like capacity to infer underlying rules from minimal examples and apply them to novel situations.
However, a significant shift occurred with the advent of advanced reasoning paradigms. Kamradt recounted that when O1 models (presumably referring to a recent major model release) first emerged, performance on ARC-AGI jumped to 21%. "That tells you something really interesting is going on," he stated. This leap indicated that AI was beginning to exhibit nascent reasoning capabilities, moving beyond sheer memorization. The community, including major labs like OpenAI, xAI (with Grok-4), Google DeepMind (Gemini 3 Pro), and Anthropic (Opus 4.5), has since recognized ARC-AGI’s utility, incorporating it into their model release evaluations.
This widespread adoption, while validating ARC-AGI's relevance, also brings a cautionary note regarding "vanity metrics." Kamradt emphasized that while the community's recognition is positive, it does not signify the completion of ARC Prize's mission. The ultimate goal is "to pull forward open AGI progress" by inspiring researchers to develop genuinely generalizable systems. Simply optimizing for a benchmark, even one as thoughtfully designed as ARC-AGI, risks falling into the trap of over-fitting without achieving true underlying intelligence. The aim is not just to pass the test, but to understand why and how the system passes it, ensuring that the progress is transferable and not just a domain-specific triumph.
To further push the boundaries, ARC-AGI is evolving. Version 1 and 2 are static benchmarks, presenting a fixed set of input-output pairs. The upcoming ARC-AGI 3.0, however, will introduce interactive, game-like environments without explicit instructions. This shift is profound. As Kamradt explained, "We're not going to give any instructions to the test taker on how to complete the environment." Instead, agents must perceive, decide, and act, receiving feedback from the environment to iteratively learn and adapt—a process that closely mirrors how humans learn in the real world. This interactivity and lack of prior instruction are designed to stress-test an AI's capacity for true human-like intelligence, demanding exploration, planning, reflection, and memory compression.
Crucially, ARC-AGI 3.0 will also introduce new metrics beyond mere accuracy. The evaluation will compare the number of actions and the amount of data (or "energy") an AI requires to beat a game against what a human needs. This directly addresses Chollet's emphasis on learning efficiency. Historically, AI has relied on "brute-force solutions" with "millions and billions of frames of video game" and corresponding actions to solve problems. ARC-AGI 3.0 will normalize AI performance to the average human performance, demanding more efficient learning.
Kamradt clarified that even if a system were to achieve 100% on ARC-AGI 1.0 and 2.0, it would be "necessary for AGI, it's not sufficient." The same holds true for ARC-AGI 3.0. A perfect score would represent the "most authoritative evidence" to date of a system capable of true generalization, triggering a deeper conversation about the nature of AGI itself. The pursuit of general intelligence, therefore, remains an ongoing journey, with ARC-AGI serving as a crucial, evolving compass guiding the path toward systems that can genuinely think and invent alongside us.
