Nathan Labenz, host of The Cognitive Revolution podcast, recently joined a16z’s Erik Torenberg for a deep dive into the frequently debated question: Is AI progress slowing down? Their discussion, prompted by Cal Newport’s observations on student reliance on AI, swiftly navigated the nuanced landscape of AI development, separating concerns about societal impact from the raw pace of capability advancement. Labenz emphasized that these are distinct questions, often conflated in public discourse.
One core insight from Labenz is the distinction between AI's impact on human behavior and its technical progress. He acknowledges Newport's concern that students might become "lazy" by offloading cognitive strain to AI. "I would cop to having exhibited myself," Labenz admitted, referring to his own tendency to prompt AI for coding solutions rather than deeply engage with the problem. This phenomenon, he notes, is a valid worry, echoing broader concerns about social media's impact on attention spans and critical thinking. However, this human-centric issue doesn't necessarily reflect a stagnation in AI capabilities.
The perception of an AI slowdown, particularly regarding models like GPT-5, is another critical point Labenz dissects. He argues that the apparent plateau in perceived performance might stem from a change in release cadence and a general acclimatization to rapid advancements. Unlike the dramatic leap from GPT-3 to GPT-4, which many users experienced as a sudden "explosion" of capabilities, subsequent models like GPT-4.5 and the eventual GPT-5 might offer improvements that are less immediately striking to the average user because many intermediate advances were released incrementally. These "boil the frog" releases, as he metaphorically describes them, might have dulled the impact of later, significant upgrades.
Labenz strongly refutes the notion that scaling laws have "petered out." He highlights that while there might not be a "law of nature" guaranteeing indefinite scaling, current models are still demonstrating substantial gains. He points to the impressive leap in a benchmark called "Simple QA," a super-long-tail trivia test, where GPT-4.5 jumped from approximately 50% to 65% accuracy. This improvement, he stresses, represents a significant gain in esoteric factual knowledge, a domain where human performance would be near zero. This is not a flatlining; it’s a continued ascent, albeit one that might be harder to quantify with a single, universally understood metric.
Beyond raw knowledge, the advancement in reasoning capabilities and context windows is a game-changer. Early GPT-4 models had an 8,000-token context window, limiting the amount of information they could process. Now, models like Gemini boast vastly longer context windows and a much-improved command of that context. This allows AI to not only digest dozens of research papers but also perform "pretty intensive reasoning over them with really high fidelity." This ability to reason over provided context, rather than solely relying on pre-trained knowledge, acts as a substitute for the model "knowing facts itself" and represents a powerful new avenue for AI utility.