Schulman still thinks the RSI clock is the outer loop

The calendar is the easy part of the recursive self-improvement argument. The hard part is who picks the next experiment.

Dwarkesh Patel sat Beren Millidge, John Schulman and Charlie O'Neill down for 97 minutes and opened with a trap. If it is 2036 and we do not have billions of superintelligences, and the reason is not a war or a ban, what is the technical stall? Millidge is CTO at Zyphra. Schulman is chief scientist at Thinking Machines, the lab that launched in February 2025 with people out of OpenAI, Meta AI and Mistral AI, and the person who ran the RLHF work that shipped ChatGPT. O'Neill runs model training at Baseten.

Millidge reached for a Moravec-style stall. Models ace hard math and chess and still fail to generalize. If a sim-to-real gap stays open, you get systems that look brilliant on the eval and weak the moment the distribution moves. He called that his default if meta-learning and continual learning stay unsolved. He also said he thinks it is unlikely, because RL already generalizes more than that story allows.

Schulman described the cycle since 2023 in the least promotional way a lab scientist can. Each new frontier model feels like AGI for a month. Then judgment and self-checking show up as the bottleneck again, and the model feels dumb. Even if it writes far more code than a person, it does not make you 100x more productive. The question is how many more of those cycles you get before research actually compounds.

O'Neill asked how far a learner on a chip really is from a global optimum. The takeoff story assumes that once an agent is 0.1% better than every human at AI research, you can run hundreds of thousands of copies and drown every other bottleneck. He compared the last decade to Moore's law: a straight line made of discrete jumps. Pre-training scaled, then hit diminishing returns. RL revived the line. The next jump may need an objective that gradient descent on this architecture cannot propose. If the missing piece is another trick you add to the current stack, a swarm of LLMs might find it. If it means throwing out gradient descent, thinking harder on transformers does not get you there.

Schulman went back to the early OpenAI intuition that next-token log loss would never buy intelligence, because the important bits were too small a fraction of the loss. Then it worked anyway. The field now lives on kinds of generalization nobody had a right to expect: from naive pre-training to deep understanding of an input, from verifiable tasks to fuzzy ones. The inner loop of post-training benchmarks still does not equal what users like.

O'Neill's sharper cut: all thinking can do is update a posterior on bits you already have. It cannot mint new bits. Give an AI the Kaplan scaling-law plot and it would have noticed the annealing mistake years earlier. muP and how learning rate scales with width are the same class of save. That is a 10x speed-up if the job is "maximize the objective we already have." It does not tell you what objective to propose next, then the next, without the loop going off the rails. That is the RSI question they kept returning to.

The last human job, Schulman said, is defining the objective. Coding the experiment is already easier than choosing it. Going from DeepMind's "solve intelligence by beating games" to Alec Radford predicting the next token on a wide scrape, then waiting for scaling laws before anyone poured compute on it, is the kind of jump agents still propose as tiny miscellaneous steps. Alignment, in that frame, is specification first and optimization second. The first half does not vanish when the model can do the technical work. A post-training team is large because someone still has to decide how the model should behave in each area.

Distillation is why the labs do not keep the pot

Patel asked why model providers do not consolidate into a handful of winners. Schulman said distillation is the main force against that. Anything you can learn through RL is a small number of bits. If you can get trajectories that show the behavior, you can copy it. Continual learning does not stop the copy. If the teacher improves every day, the student can distill every day.

Prompt distribution is the hard part. Even with full access and chain of thought, supervised distillation is non-trivial because you have to ask the teacher the right things, on a wide and realistic set of prompts. Millidge said Chinese labs are likely using router services that let people in China hit US frontier models, mostly for coding, and that those routers collect and sell the traces. That is the prompt distribution you cannot invent. Seed prompts still come from humans and that kind of data. Then models synthesize coverage. Humans provide fewer bits as the models get better. A live user saying "that didn't work, add this feature, actually step back" is still a different object than a synthetic "I want traces that look like this."

The concrete prediction from that stack is ugly for winner-take-all stories. Millidge said GLM-5.3 and Kimi K3 now look, to him, almost objectively better than Sonnet 5 and Opus 5, even though the US labs have had distillation and even logit distillation from Mythos. If you have the hard RL environments and the logits and you still lose, either real-world deployment matters more than the environment, or the student-teacher gap is in an uncanny valley. Schulman split the environment axes into difficulty and realism. Benchmaxxing is the easy axis: hard, verifiable, puzzle-like. Realistic multi-turn work with a human in the loop is the other. Distill naively and you match the teacher on the first axis. Miss the second and the student looks great on the leaderboard and worse in the agent setting. Big models generalize from narrow hard tasks to realistic ones. Small students need the realistic prompt distribution or they do not get those behaviors.

The Chinese labs can also buy the same labeled data the frontier labs buy from the same vendors. Keeping up is not supposed to be this easy. It is.

How the first RSI-capable models get trained is less romantic than a GPU and a blank notebook. Schulman said it will be human feedback to absorb researcher taste, plus multi-step research practice environments, patched each generation for whatever was most broken last time. O'Neill's picture: you do not roll the lineage back to before GRPO and ask the model to rediscover RL. You stay at the frontier, turn the last three months of bugs and improvements into environments, and distill that back in. It feels asymptotic because you are always chasing what the humans just found. The escape hatch is that environments can go above humans. A nanochat speedrun faster than any human, or a loss target no person can hit, is a legal goal. A lot of real research is not that. It is an intuition, a slightly relaxed task to show signs of life, then a walk back toward realism. Whether that generalizes enough for the loop to seal without a human is the open question.

The lab bet, as Patel restated it and they did not really reject, is scale RLVR across millions of environments until persistence, triage, and long-horizon agency fall out, then deploy that as a drop-in remote worker, still learning meta-skills in the data center rather than from live weight updates. Anthropic's path is the exhibit: coding first, then finance and Excel, then PowerPoints, the long tail of the working economy. If models were already good at learning on the job, baking PowerPoint into the weights would be a waste. They are not there yet. Domain data is also a way to buy revenue for the next run, and there is not that much direct RSI data in the world. Schulman added the efficiency case: even a model that could read the books on the fly might still want the intuitions baked in. Sim-to-real stays dominant while sample efficiency is low, because no human will sit in the RL loop for thousands of turns.

About half of compute is inference that does not write back into the same weights. Patel called that the missing hive mind: millions of years of deployment that should, eventually, land in one model. They said the coarse version is already happening in the next generation's mid-training, especially if you filter and annotate. The Chinese labs, they said, do this without apology. Composer and Harvey already turn deployment complaints into environments and post-train a Kimi-class model on a faster cadence. Cursor's older Tab model updated from accept or reject. They later did a similar REINFORCE loop on the generative model, with heuristics to estimate better-or-worse-than-average, and shipped every five hours if CursorBench went up. Superficial rewards get hacked. Accept-the-edit is one of those.

The sample-efficiency gap is the stall Patel keeps in his pocket. Models may be a millionfold behind a human childhood in data-per-capability. If sim-to-real transfer is weak on year-long work that includes a client, a court, or a trading day, you need weight updates from the real thing, and you do not have the sample efficiency to take them. Millidge split tasks into cumulative and non-stationary. RSI, in the hopeful version, is a Python file: attention, mixture of experts, GRPO, each a line you keep. A legal associate is the other kind. Relationships move. Implicit process moves. It is unfortunate, he said, that RSI may be easier than being a paralegal.

Taste, for Schulman, is partly behavior that works in the long run: what stays maintainable in a software project. Dump a trillion tokens of one researcher's life into context and you still have to train the model to make the right update from that context. Humans grow taste on short careers. A PhD is five years and maybe 10 to 30 projects. Models will have more episodes. Whether that meta-learning reaches really long horizons is unsolved.

A shared hive mind that trains on everyone's deployment is also an incentives problem. Companies will not want the provider to eat their advantage. The economic path is modules first: LoRAs, compressed KV cartridges, linear attention, something you can sub in without rewriting the base. Traces still leak into the next pre-train on a three-month cadence, then a week, then a day, then an hour. That is not the same as a live weight update on one shared brain.

Zoom into one law firm and the methods break. SFT or on-policy distillation on successful traces, repeated as hundreds of micro-updates, forgets earlier information and decays general capability. RL changes very little, which is why it is safe and why it is bad at injecting "this person does this, here is the process." Capacity is not the limit. Train the same-size model from scratch with all the data in mid-training and it is better. Plasticity and a shifting data distribution are the limit. That is expensive, and it is why people still cut a new base.

The clocks they were willing to say

On a drop-in remote worker for white-collar work, not only coding, with a month of seamless computer use: O'Neill said about a year if the firm makes itself programmatically accessible, a couple of years if you mandate a browser. Millidge said three years for full generality, with 80 to 90 percent earlier once companies rebuild around the models. Schulman noted that some Upwork-grade remote work is already worse than current AI, and that we will keep moving the goalposts down a long tail. Millidge's tax example: he told Codex to collect everything and send it to the accountant, including a pile of clicking and downloading, and it did.

On a 10x uplift for AI researchers themselves, not generic knowledge work: Charlie said two years. Millidge could see two years if the model can run even one or two experimental loops without crashing, because coding is already past 10x. An earlier five-to-ten-year instinct did not survive the clarification that this was researchers, not a general worker. Patel's follow-up is the obvious one. Plug a 10x researcher speed-up into a naive model of AI progress and the pace changes a lot, until it bottlenecks on something else. Millidge's crux is still his own capacity to absorb the result and pick the next experiment. If the model can run two or three in a row, that is already the uplift.

On an AI that beats top humans across every computer-based field, including work that takes years: O'Neill said three to four years. Schulman said five to ten, and treated automating AI research as close to ASI-complete, especially anywhere that needs longer-horizon learning than a million-token window plus notes. Millidge landed near five years for the domains labs actually fund, with a long tail of fields nobody has allocated compute to. Spatial and mechanical work, and jobs like becoming superhuman at TSMC, sit on that tail. The model may not need to learn a new domain as fast as a human. It will have more experience than any human.

That is why 2036 is a weak headline. It asks whether the next scale jump arrives on schedule. The conversation on the tape is about distillation stealing the easy axis, micro-updates destroying the base, and whether anyone has a procedure for choosing the jump. They were willing to put years on a worker and a researcher. They were not willing to say the outer loop is solved.