Llion Jones, a co-inventor of the ubiquitous Transformer architecture, contends that the artificial intelligence industry is ensnared in a "local minimum" of its own making. This provocative argument, delivered alongside Sakana AI research scientist Luke Darlow, challenges the prevailing dogma that more scale and iterative tweaks to existing models will unlock genuine intelligence. Instead, Sakana AI proposes a biologically inspired paradigm shift with their Continuous Thought Machines (CTM).
Jones and Darlow spoke with the Machine Learning Street Talk podcast about the industry's "success capture," a phenomenon where the remarkable efficacy of a dominant technology, like the Transformer, inadvertently stifles fundamental innovation. Llion Jones, having spent years deeply entrenched in Transformer research, made the deliberate decision to step away. "I'm going to drastically reduce the amounts of research that I'm doing specifically on the Transformer because of the feeling that I have that it's an oversaturated space," he stated, highlighting a personal conviction that the path to true intelligence lies elsewhere.
The core of Sakana AI's critique revolves around the distinction between mimicry and genuine understanding. Jones illustrates this with a striking "spiral problem" analogy: if a standard neural network is tasked with understanding a spiral, it typically approximates it with numerous tiny straight lines, effectively "faking" the shape without grasping the underlying concept of spiraling. Today's AI models, despite their impressive capabilities, similarly excel at mimicking intelligent responses without possessing an internal process of genuine "thinking" or reasoning. They are brilliant imitators, but lack the ability to truly change their minds or backtrack when faced with a complex problem.
Sakana AI’s proposed antidote is the Continuous Thought Machine (CTM), a novel architecture fundamentally inspired by biological brains. Luke Darlow explained that unlike traditional AI, which might attempt to solve a maze by "staring at the whole image and guessing the entire path instantly," the CTM "walks" through the maze step-by-step. This sequential, iterative process allows the CTM to naturally spend more time on challenging problems, effectively "pondering" solutions and correcting its own mistakes, a crucial leap towards more human-like reasoning. CTM’s behavior is rooted in a new kind of representation: the synchronization between neurons over time.
This pursuit of genuinely novel architectures is deeply embedded in Sakana AI's organizational philosophy, which champions research freedom. Llion Jones nostalgically recalled the origins of the Transformer: "The Transformers was very, very bottom up... it was a bunch of people talking over lunch... having the freedom to have, you know, literally months to dedicate to just trying this idea." This contrasts sharply with the current corporate AI research landscape, where immediate commercialization pressures often dictate research agendas, leading to what the podcast host termed "technology capture."