Poolside's Eiso Kant on building AI models in 8 weeks

In a recent episode of the Latent Space podcast, Eiso Kant, co-founder and CEO of Poolside AI, shared insights into the company's ambitious approach to building foundation models. Kant, who has spent over a decade focusing on language models for code, believes that code is the key to achieving Artificial General Intelligence (AGI).

The Poolside Model Factory

Kant elaborated on Poolside's 'Model Factory,' an engineering system designed for rapid iteration and deployment. This system handles everything from pre-training to final model release, enabling the company to iterate on models in as little as eight weeks. The factory supports a massive scale of experimentation, with tens of thousands of experiments running monthly. Key features include streaming data directly into training, ensuring reproducible experimentation, leveraging low-precision compute, and utilizing agents that can autonomously write code, launch jobs, evaluate results, and modify the very pipelines used to train future models.

This approach is a significant departure from traditional foundation model training, which often requires substantial manual intervention and lengthy cycles. Kant highlighted that Poolside's journey began with a conviction that code was the path to AGI, even when the market wasn't ready. This conviction led to significant investment, including $12 million spent over four years on an idea before it gained traction.

Openness and Competition

A core tenet of Poolside's philosophy is embracing open weights and open research. Kant expressed a preference for a world with 100 foundation model companies rather than a concentrated market dominated by a few, even if Poolside were among the dominant players. He believes this openness fosters innovation and healthy competition, ultimately benefiting the entire field.

Laguna S and the Future of AI

Kant also discussed Poolside's model 'Laguna S,' which features 118 billion total parameters with 8 billion active parameters. He emphasized that persistence, verification, and backtracking might be more crucial than raw intelligence in achieving advanced AI capabilities. Furthermore, he touched upon the potential of smaller models, the increasing role of reinforcement learning in pre-training, and the idea that current next-token prediction methods may not be extracting enough value from the vast amount of data available on the web.

The conversation also touched on the practical aspects of scaling models, the economics of training, and the underlying hardware systems, such as those provided by Nvidia and TSMC, that power this progress. Kant also shared the story behind the company's name, 'Poolside,' and how it reflects their refusal to compromise on ambition.