Why the AI Base Model Is Changing: Varun Singh on Data Recipes

The traditional artificial intelligence base model built purely on general web text scrapes is reaching the end of its era. At the AI Engineer World's Fair, Varun Singh, pre-training lead at Arcee AI, laid out why the old approach to training large language models no longer meets the needs of modern AI reasoning engines. Instead of serving as a simple repository of human internet knowledge, base models are being re-engineered from the ground up to act as specialized foundational priors for reinforcement learning and agentic workflows.

Who Is Varun Singh

Varun Singh leads pre-training initiatives at Arcee AI, a venture focused on specialized open model architectures and domain adaptation. His work concentrates on optimizing pre-training data mixes, synthetic data generation, and context extension methods for domain-specific models like Arcee Trinity Large Thinking. Singh brings hands-on experience in balancing complex multi-stage training pipelines across both dense and Mixture-of-Experts (MoE) neural architectures.

The Fall of Web Text in Pre-Training Mixes

For years, pre-training pipelines relied almost entirely on massive web dumps. Common Crawl and WebText made up roughly 85 percent of the pre-training data mix for OpenAI's GPT-3, with general human knowledge serving as the core foundation for next-token prediction. By the time Meta released Llama 3, general knowledge tokens still made up about 50 percent of the mix. However, recent releases demonstrate a drastic shift away from passive web scrapes.

Singh pointed out that modern pre-training recipes, such as MAI-Base-1, have reduced web text to just 15 percent of total tokens. Data mixes are now heavily weighted toward code, mathematics, and technical STEM documents. "Web text, which used to make up like up to 85 percent of the training data in GPT-3, is now all the way down at 15 percent," Singh noted. "That just shows that the value of web text contributing to downstream performance is taking a backseat to things like code and STEM abilities."

Reinforcement Learning Takes Center Stage

In early language model development, reinforcement learning (RL) was viewed as a minor alignment step to adapt models to chat templates. The release of OpenAI's o1 and DeepSeek's R1 changed that dynamic completely. Reinforcement learning is now a heavy compute consumer that directly builds advanced reasoning capabilities instead of merely polishing conversational tone.

Labs are shifting compute allocation significantly toward post-training RL phases. Research figures show that modern reasoning systems spend an equal or greater proportion of their compute budget on RL compared to initial supervised pre-training. Models like Composer 2.5 push this dynamic further by pouring vastly more compute into RL exploration than into supervised learning. According to StartupHub.ai data, model families like GLM carry a score of 3/100 on our index, while adjacent model infrastructure providers such as Seldon (58/100), Poetiq (52/100), Mirai (51/100), Zymtrace (50/100), and Axiomatic AI (50/100) highlight the growing commercial focus around execution toolchains built for these multi-stage RL pipelines.

Synthetic Data and MoE Load Balancing

To prepare base models for intense post-training RL, AI labs are moving post-training datasets and synthetic content into the early pre-training stage. In Arcee AI's Trinity Large model, web-scale synthetic data is added through rephrasing techniques. Seed data items are upsampled by generating synthetic rephrases of the same information, forcing the network to internalize concepts across multiple linguistic variations without collapsing performance.

Pulling post-training instruction data directly into early pre-training also resolves severe architectural hurdles in MoE models. When an MoE model encounters a radically different data distribution during post-training fine-tuning, expert routing becomes severely imbalanced. By introducing Supervised Fine-Tuning (SFT) sequences during initial pre-training, as seen in Nvidia (NASDAQ:NVDA)'s NeMoTron 3 Ultra, experts specialize evenly from the beginning of the run, avoiding brittle load-balancing hacks later in the pipeline.

Supervised Learning as a Prior for RL

Rather than treating pre-training, mid-training, and post-training as isolated silos, research labs increasingly view all initial next-token prediction work as supervised pre-conditioning for reinforcement learning. The objective of supervised learning is no longer to memorize the internet, but to instill atomic problem-solving primitives that the model can combine and extrapolate when exploring complex environments during RL.

Singh concluded that as software agents and reasoning models become the primary interface for software interaction, base model engineering must adapt. The goal of early training is no longer building a passive encyclopedia of facts, but crafting a targeted structural prior that allows reinforcement learning algorithms to produce reliable reasoning behaviors.