The race towards Artificial General Intelligence (AGI) lacks clear benchmarks. Google DeepMind aims to address this with a new cognitive framework. Their paper, "Measuring Progress Toward AGI: A Cognitive Taxonomy," lays out a scientific approach to evaluating AI's general intelligence.
The framework draws from psychology and neuroscience, identifying 10 core cognitive abilities crucial for AGI. These include perception, generation, attention, learning, memory, reasoning, metacognition, executive functions, problem-solving, and social cognition.
Deconstructing Intelligence
Each ability is defined based on extensive cognitive science research. The goal is to create empirical tools to track AI development beyond task-specific performance.
The proposed evaluation protocol involves benchmarking AI systems against human capabilities across these cognitive tasks. This method aims to prevent data contamination and provide a relative measure of intelligence.
To move from theory to practice, DeepMind is partnering with Kaggle for a hackathon. The event focuses on building evaluations for five critical abilities: learning, metacognition, attention, executive functions, and social cognition. This initiative is a key part of advancing AGI evaluation framework development.
Researchers can use Kaggle's Community Benchmarks platform to test their evaluations against leading AI models. The hackathon offers a substantial $200,000 prize pool, with awards for top submissions across five tracks and grand prizes for the best overall entries.
Submissions are open from March 17 to April 16, with results announced on June 1. This effort by Deepmind seeks to foster community-driven progress in defining and measuring AGI.
