#Pre-training
4 articles with this tag
UltraX: Redefining LLM Data Refinement
UltraX redefines LLM data refinement by introducing function-calling for fine-grained editing, achieving superior performance with fewer training tokens.
How Andrej Karpathy's Software 2.0 Thesis Became Software 3.0
Karpathy's 2017 'Software 2.0' essay predicted that neural networks would absorb entire engineering stacks. His 2026 Sequoia Ascent talk defines a third era, where large language models automate anything humans can verify. Here is how the thesis evolved across nine years and four organizations.
Andrej Karpathy Joins Anthropic to Lead Claude Pre-training Research
Andrej Karpathy joined Anthropic on 19 May 2026 to lead a new team using Claude to accelerate pre-training research, closing a 22-month chapter at Eureka Labs.
Pre-training Space RL for Enhanced LLM Reasoning
New PreRL framework optimizes LLM reasoning by directly refining the pre-training distribution P(y), enhanced by Negative Sample Reinforcement and Dual Space RL.