"Garbage in, garbage out," states Elie Bakouch, who leads pre-training efforts at Hugging Face and is a key architect behind SmolLM. This seemingly simple adage encapsulates a profound shift in the development of large language models: the era of simply scaling models and data to astronomical sizes is yielding to a more sophisticated, multi-faceted approach focused on optimization and efficiency. The relentless pursuit of larger models, while once the primary driver of progress, is now complemented, if not superseded, by a deep dive into the foundational sciences of model training.
Companies working on this
Profiles of the companies named in this story, with founding year, headquarters, and a short description from our database.
Hugging Face is the leading AI community and platform for machine learning collaboration, enabling developers to build, share, and deploy models, datasets...
- Founded
- 2016
- Location
- New York, United States
- Valuation
- $4.5B
Bakouch recently spoke with Alessio Fanelli and Swyx on the Latent Space podcast, offering a revealing glimpse into Hugging Face's research philosophy and the intricate mechanics behind their latest innovations. The conversation centered on Bakouch's "unified view of model training," a framework comprising five interdependent pillars: data quality optimization, model architecture design, information extraction efficiency, gradient quality maximization, and training stability at scale. This holistic perspective underscores that achieving state-of-the-art performance in LLMs is no longer a singular challenge but a delicate balancing act across numerous engineering and scientific frontiers.
The first and arguably most critical pillar, data quality optimization, emphasizes the paramount importance of high-quality and diverse datasets. Hugging Face's work on open science data, including FineWeb-Edu2 and the recently released FinePDF dataset, exemplifies this commitment. FinePDF, for instance, offers 3 terabytes of meticulously curated, high-quality data extracted from PDFs, demonstrating superior performance over generic web datasets.
Beyond data, Bakouch delved into the evolving landscape of model architectures and training methodologies. He highlighted the limitations of long-standing optimizers like Adam when applied to increasingly massive models, noting that "Adam parameters for LLaMA2 are not optimal for mega-sized models." This observation has spurred innovation in gradient quality maximization, leading to alternatives such as Muon and Shampoo. Muon, in particular, is lauded for its enhanced stability and capacity to explore new solution spaces, often paired with techniques like QK-Clip to prevent "exploding attention logits" during early training.
A significant portion of the discussion revolved around Mixture of Experts (MoE) architectures, which Bakouch emphatically declared "MoE are faster to train!" This assertion stems from their ability to achieve comparable performance to dense models with fewer computational FLOPs during training and significantly reduced inference costs. MoE models achieve this by activating only a sparse subset of their total parameters, leveraging specialized "experts" for different tasks or data types. This inherent sparsity is a game-changer for the inference landscape, allowing for larger, more capable models to run efficiently on more constrained hardware.