AI Research

Agents beat humans on speedruns, still can't invent

Prime Intellect pitted Claude Code and Codex against humans on the NanoGPT optimizer speedrun. They won on steps, but produced no new optimizer.

StartupHub.ai Staff

By StartupHub.ai Staff, reviewed by Daniel Singer, Editor

4 min read

Prime Intellect let Claude Code and Codex race human researchers on a NanoGPT speedrun and the agents won on time, per AI Engineer.

Agents beat humans on speedruns, still can't invent - AI Engineer
Agents beat humans on speedruns, still can't invent, AI Engineer

Research engineer Elie Bakouch described the setup as a test for recursive self-improvement that labs can run in the open. The game is simple. Reproduce GPT-2 validation loss in the shortest wall-clock time, using the same data. It started with Andrej Karpathy training GPT-2 from scratch in 90 minutes, then community work led by Keller Jordan squeezed that to under two minutes. That history made the speedrun a tight, verifiable benchmark.

The wrinkle is which speedrun you play.

The original NanoGPT track lets you change almost anything. The newer optimizer speedrun, released a few months before the talk, restricts changes to optimizer-related parameters only, such as swapping Adam for Muon or Shampoo. That constraint is what makes it researchy. You cannot just engineer faster kernels. You have to find a better method.

Prime Intellect launched two agents about two months before the talk. Codex, running as GPT-5.1 with extended thinking, and Claude Code, running as Opus 4.5 with extended thinking, were given a minimal agent loop on a SLURM cluster with preemptable permissions, a scratch pad for memory, and the ability to sbatch jobs, read logs, and claim a record only after a statistical threshold. Reward was binary. Beat the last record and you score. Miss and you get zero.

The behaviors split fast. Claude Code kept pausing every nine or ten hours to declare it could not improve, leaving it idle about a third of the time until Bakouch nudged it to keep exploring. Codex never stopped. It also wrote far more to the scratch pad, spawned more sub-agents, burned more tokens, and compacted context far more often. Claude Code logged one compaction for the whole run. Codex did about 20 per hour. The tone differed too. Claude Code celebrated records with emoji. Codex read like a lab notebook.

On score, both beat humans most of the time. Humans were white on Bakouch's chart. Claude Code was orange, Codex blue. Claude Code jumped early. Codex closed steadier. At the time of the freeze, the best human mark was around 2,990 steps to reach the target loss. Claude Code beat it by 50 to 60 steps. Codex beat it by about 20. A restart trick helped. When Bakouch restarted Claude Code a day or two before release, it fetched the latest human records and improved on them. That live retrieval is part of the design for a full-access track.

That design is why Prime Intellect is now rebuilding the benchmark properly. Bakouch said the first run was fun but unstructured. The next version will run multiple seeds and three access modes: no external knowledge, arXiv only, and full access including latest human records, on both the NanoGPT and optimizer tracks. Each run will take 15 to 20 minutes, which makes it fast enough to use as a reward for training as well as evaluation.

A preview on the optimizer track over almost five to six days shows the same pattern at larger scale. Claude Code improved progressively, Kimi K2 Thinking made a step-function breakthrough on day four to beat Codex for the lead, and a GLM run was still iterating at talk time. When you normalize by output tokens, the cost story flips. Kimi was far more token efficient than Claude Code in max mode. Codex sat in the middle.

The impact lands on how research teams budget and operate. If agents can reliably shave steps by combining existing tricks but cannot yet invent a new optimizer, workflows shift toward sandboxed GPU loops where humans steer.

Bakouch was blunt about the ceiling. He expected agents to propose wild new optimizers. They did not. They combined papers and delivered plus-one gains. One Claude Code run even surfaced a paper no other model found and that paper helped produce the best record, but it was still recombination, not invention. His proposed fix borrows from Google's AlphaEvolve. Use many generators, including cheaper open models, propose ideas, run the speedrun for reward, have a judge score quality and taste, then scale promising methods to larger parameter and token counts where many speedrun tricks fail to transfer.

Prime Intellect is building that loop now: GPU sandboxing so agents can iterate safely, an agent framework with a real file system and programmatic tooling, and verifier libraries for training and evaluation. The novelty track that requires a record with only novel ideas remains the hard one, and that is the point. Beating a timer is solved. Inventing the next method is not.

Mentioned in this story

Markets

Sources

Get the free daily brief every morning

Every morning the day's AI news, new AI startups and new funding rounds. Free and takes a few minutes to read, unsubscribe at any time.

Join 64,000 other subscribers.

TopicsPrime IntellectClaude CodeCodexElie BakouchAndrej KarpathyKeller JordanMuonKimi

StartupHub.ai Staff

Written by

StartupHub.ai StaffEditorial team

The staff writers of StartupHub.ai, ranging from investment analysts to avid AI tool users, early adopters and critical enthusiasts. Backgrounds span engineering, business and the arts. We hold every piece to rigorous standards of research and review.

Daniel Singer

Reviewed by

Daniel Singer is an AI expert who has been writing about, covering and building AI technology since 2018.

Test your Agent Readiness now

Paste your website. How well ChatGPT, Claude and the other agents can read and use a site, in about a minute.

Watch

The videos behind the news, one flip at a time

The feed mixes the videos from our articles with startup demos and you flip through them like TV channels for free with no signup.