# UC Berkeley PhD Student Challenges AI Evaluation Methods _Parth Asawa, a PhD student at UC Berkeley, argues that current AI evaluation methods are insufficient for measuring continual learning and calls for a new benchmark approach._ **Published:** 2026-08-12 **Source:** https://www.startuphub.ai/ai-news/ai-research/2026/uc-berkeley-phd-student-challenges-ai-evaluation-methods --- At the AI Engineer World's Fair, Parth Asawa, a PhD student at UC Berkeley, presented a critical look at how artificial intelligence, particularly large language models (LLMs), are evaluated. Asawa argued that the current approach, which often treats each task in isolation, fails to measure a crucial aspect of intelligence: the ability to learn and adapt over time, a concept known as continual learning. Parth Asawa (UCB PhD)Core UC Berkeley PhD student challenges current AI evaluation methods for LLMsFrom the articleAt the AI Engineer World's Fair, Parth Asawa, a PhD student at UC Berkeley, presented a critical look at how artificial intelligence, particularly large language models (LLMs), are evaluated.Rethink EvaluationContextcritical look at how large language models are currently evaluatedFrom the article 3 mentionsAsawa highlighted how the typical evaluation of LLMs involves presenting them with a series of independent tasks.AI Engineer FairCoreFrom the articleAt the AI Engineer World's Fair, Parth Asawa, a PhD student at UC Berkeley, presented a critical look at how artificial intelligence, particularly large language models (LLMs), are evaluated.critiquesCurrent AI BenchmarksDriverevaluate LLMs on isolated tasks, ignoring past experiences and learningFrom the article 5 mentionsAsawa proposed three key criteria for designing effective continual learning benchmarks:leads toFails Continual LearningDriverthis method cannot measure how models learn and adapt over timeFrom the article 9+ mentionsAsawa argued that the current approach, which often treats each task in isolation, fails to measure a crucial aspect of intelligence: the ability to learn and adapt over time, a concept known as continual learning.requiresNeed New BenchmarkEffectcalls for a new approach to truly assess models' learning abilityFrom the article 4 mentionsShared Structure: To enable improvement from prior experience, tasks need to have underlying shared latent structures that models can exploit.definesTrue Learning MeasureContextFrom the article 3 mentionsHe contrasted this with what a true measure of learning ability should look like: performance that improves as a function of prior experience, rather than a scattered, inconsistent performance. ## The Limitations of Current AI Benchmarks Asawa highlighted how the typical evaluation of LLMs involves presenting them with a series of independent tasks. This method, he explained, is akin to asking a model to restart from scratch every single time, fundamentally ignoring its past experiences and learning capabilities. This approach is insufficient for assessing how well models can truly learn and improve from prior interactions. He contrasted this with what a true measure of learning ability should look like: performance that improves as a function of prior experience, rather than a scattered, inconsistent performance. Asawa defined continual learning as **"sample efficient, online learning that is stable over long horizons."** This involves retaining information without forgetting while simultaneously adapting to new data. ## Rethinking Continual Learning Evaluation The current paradigm in LLM training involves offline processes where models are trained on vast datasets and then deployed as frozen checkpoints. Continual learning aims to change this by enabling models to learn and update their parameters over time. Asawa discussed various approaches being explored, including in-context learning, external memory stores, and updating model weights directly. However, his core argument was that the field is not adequately evaluating these capabilities. He stressed that if continual learning is not about "point capabilities," then the evaluation methods must change to reflect this. Asawa proposed three key criteria for designing effective continual learning benchmarks: - **Headroom:** Tasks must require online adaptation or learning, meaning models shouldn't be able to perform well by simply relying on offline training on that specific data. - **Shared Structure:** To enable improvement from prior experience, tasks need to have underlying shared latent structures that models can exploit. - **Learning Mechanism:** There must be a signal within the environment, such as rewards, error messages, or textual feedback, that guides the model's learning process. Asawa also elaborated on the metrics used for evaluation, emphasizing the importance of the **'gain' metric**. Gain is calculated as the difference between stateful reward (where the model can maintain state) and stateless reward (where the model is reset between tasks). This metric helps to isolate the actual benefit derived from accumulated experience, rather than just reflecting the base model's initial capability. ## The Continual Learning Bench 1.0 and Failure Modes Asawa introduced the [Continual Learning Bench 1.0](/ai-news/ai-video/2025/terminal-bench-2-0-and-harbor-reset-the-bar-for-ai-agent-evaluation), a benchmark suite featuring tasks from six different domains, including signal processing, software engineering, epidemiology, game playing, database exploration, and sales prediction. Each task is designed with specific reward metrics and validated by domain experts to ensure realism and learnability. He then discussed observed failure modes in continual learning, categorizing them into stability and plasticity issues. Stability failures occur when models fail to retain new information, while plasticity failures happen when models cannot adapt to new information. He provided examples from the sales prediction task (demonstrating a stability failure) and the epidemiology task (illustrating a plasticity failure) to highlight these concepts. ## Challenging the Status Quo in AI Research Reflecting on broader implications, Asawa suggested that the current LLM training stack, which has evolved over years, might not be optimally designed for continual learning. He posed the question of whether the field is falling into a sunk cost fallacy by trying to adapt existing methods rather than redesigning models from the ground up with continual learning as a primary objective. Furthermore, Asawa called for a re-evaluation of how AI research is conducted as a whole, touching upon issues of open science, power consolidation, and safety. He encouraged greater involvement in reimagining the future of AI research institutions and open science practices. The presentation concluded with a summary of the key takeaway: **"Continual learning doesn't look like point capabilities. We need to measure it the right way to optimize for the right objective as a field."** This sentiment underscores the need for a fundamental shift in how AI’s learning abilities are assessed and developed. --- Original analysis from [startuphub.ai](https://www.startuphub.ai), the #1 AI startup directory.