#Pretraining

4 articles with this tag

Why the AI Base Model Is Changing: Varun Singh on Data Recipes
AI Research

Why the AI Base Model Is Changing: Varun Singh on Data Recipes

Varun Singh of Arcee AI explains how web text is dropping from pre-training data recipes as AI labs shift focus toward reinforcement learning priors.

4 months ago
Andrej Karpathy's Nano Repos: 120,000 Stars and One Education Thesis
AI Figures

Andrej Karpathy's Nano Repos: 120,000 Stars and One Education Thesis

Andrej Karpathy's nanoGPT, nanochat, and micrograd have crossed 120,000 GitHub stars. Two years after founding Eureka Labs, the nano* series explains why Anthropic hired him to lead pretraining research.

4 months ago
Pretraining's Hidden Experts: A New Post-Training Paradigm
AI Research

Pretraining's Hidden Experts: A New Post-Training Paradigm

Large pretrained models are dense with task-experts, enabling simple random sampling and ensembling to rival complex post-training AI optimization methods.

8 months ago
AI Learns Beyond Text
Artificial Intelligence

AI Learns Beyond Text

AI is moving beyond text, with multimodal pretraining enabling models to learn from images, audio, and video for richer comprehension.

8 months ago