A $294,000 training run with no human-labelled reasoning data produced DeepSeek-R1, the paper that subsequently reached the cover of Nature. Liang Wenfeng, co-founder and CEO of DeepSeek, is listed as the corresponding author. The paper, published on arXiv in January 2025 under the title DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, demonstrated that a language model trained on pure reinforcement learning signals, without any supervised fine-tuning on human-curated reasoning chains, could produce world-class mathematical and coding reasoning.
Training without a teacher: how GRPO replaced supervised fine-tuning
The standard large language model training recipe in 2024 followed three stages: pre-train a base model on large text corpora; fine-tune it on human-curated examples with step-by-step reasoning (supervised fine-tuning, or SFT); then apply reinforcement learning, typically Proximal Policy Optimization (PPO), to align outputs further. The SFT stage for reasoning required expensive chains of thought labelled by humans or distilled from a more capable teacher model.
DeepSeek-R1-Zero, the experimental variant described in the paper, removed the SFT stage entirely. Starting from DeepSeek-V3-Base, the team applied reinforcement learning directly using a custom algorithm, Group Relative Policy Optimization (GRPO). GRPO's structural advantage over PPO is that it does not require a separate evaluator model of the same scale as the policy model. Instead, it samples a group of responses and uses relative performance within that group as the reward signal. As the paper states, GRPO "directly estimates the baseline from the group scores," cutting the compute footprint of the RL stage substantially. The reward signal itself was minimal: positive feedback for a correct final answer on math or coding problems, and for adherence to the required output format (thinking wrapped in tags, final answer in a separate tag). No partial credit, no reasoning-quality scoring, no human feedback at any stage.
The cost implication is direct. DeepSeek-V3's training was reported by 36kr to have run to approximately $5.6 million; the R1 training run cost $294,000, per the same reporting citing the DeepSeek team. The savings came from removing the SFT stage and eliminating the need for a matching-scale evaluator in the RL phase.
The 'aha moment' and what Nature's editors recognised
The most notable result in the paper was not the cost reduction. It was the behaviour the model developed without being instructed to. DeepSeek-R1-Zero, during RL training, began allocating more reasoning tokens to harder problems and started explicitly noting within its chain of thought when an earlier step was incorrect and correcting it mid-chain. The paper documents the training checkpoint where this first occurs as an "aha moment." The four emergent behaviours, none of them specified in the reward function, were: self-reflection, self-verification, dynamic strategy adaptation, and active exploration of alternative solution approaches.