Agents beat humans on speedruns, still can't invent

Prime Intellect let Claude Code and Codex race human researchers on a NanoGPT speedrun and the agents won on time, per AI Engineer.

Research engineer Elie Bakouch described the setup as a test for recursive self-improvement that labs can run in the open. The game is simple. Reproduce GPT-2 validation loss in the shortest wall-clock time, using the same data. It started with Andrej Karpathy training GPT-2 from scratch in 90 minutes, then community work led by Keller Jordan squeezed that to under two minutes. That history made the speedrun a tight, verifiable benchmark.

The wrinkle is which speedrun you play.

The original NanoGPT track lets you change almost anything. The newer optimizer speedrun, released a few months before the talk, restricts changes to optimizer-related parameters only, such as swapping Adam for Muon or Shampoo. That constraint is what makes it researchy. You cannot just engineer faster kernels. You have to find a better method.

Prime Intellect launched two agents about two months before the talk. Codex, running as GPT-5.1 with extended thinking, and Claude Code, running as Opus 4.5 with extended thinking, were given a minimal agent loop on a SLURM cluster with preemptable permissions, a scratch pad for memory, and the ability to sbatch jobs, read logs, and claim a record only after a statistical threshold. Reward was binary. Beat the last record and you score. Miss and you get zero.

The behaviors split fast. Claude Code kept pausing every nine or ten hours to declare it could not improve, leaving it idle about a third of the time until Bakouch nudged it to keep exploring. Codex never stopped. It also wrote far more to the scratch pad, spawned more sub-agents, burned more tokens, and compacted context far more often. Claude Code logged one compaction for the whole run. Codex did about 20 per hour. The tone differed too. Claude Code celebrated records with emoji. Codex read like a lab notebook.

On score, both beat humans most of the time. Humans were white on Bakouch's chart. Claude Code was orange, Codex blue. Claude Code jumped early. Codex closed steadier. At the time of the freeze, the best human mark was around 2,990 steps to reach the target loss. Claude Code beat it by 50 to 60 steps. Codex beat it by about 20. A restart trick helped. When Bakouch restarted Claude Code a day or two before release, it fetched the latest human records and improved on them. That live retrieval is part of the design for a full-access track.

That design is why Prime Intellect is now rebuilding the benchmark properly. Bakouch said the first run was fun but unstructured. The next version will run multiple seeds and three access modes: no external knowledge, arXiv only, and full access including latest human records, on both the NanoGPT and optimizer tracks. Each run will take 15 to 20 minutes, which makes it fast enough to use as a reward for training as well as evaluation.

A preview on the optimizer track over almost five to six days shows the same pattern at larger scale. Claude Code improved progressively, Kimi K2 Thinking made a step-function breakthrough on day four to beat Codex for the lead, and a GLM run was still iterating at talk time. When you normalize by output tokens, the cost story flips. Kimi was far more token efficient than Claude Code in max mode. Codex sat in the middle.

The impact lands on how research teams budget and operate. If agents can reliably shave steps by combining existing tricks but cannot yet invent a new optimizer, workflows shift toward sandboxed GPU loops where humans steer.

Bakouch was blunt about the ceiling. He expected agents to propose wild new optimizers. They did not. They combined papers and delivered plus-one gains. One Claude Code run even surfaced a paper no other model found and that paper helped produce the best record, but it was still recombination, not invention. His proposed fix borrows from Google's AlphaEvolve. Use many generators, including cheaper open models, propose ideas, run the speedrun for reward, have a judge score quality and taste, then scale promising methods to larger parameter and token counts where many speedrun tricks fail to transfer.

Prime Intellect is building that loop now: GPU sandboxing so agents can iterate safely, an agent framework with a real file system and programmatic tooling, and verifier libraries for training and evaluation. The novelty track that requires a record with only novel ideas remains the hard one, and that is the point. Beating a timer is solved. Inventing the next method is not.