World Models: The Key to AGI?

The quest for Artificial General Intelligence (AGI) hinges on solving a fundamental challenge: sample efficiency. How can AI models learn new tasks and skills rapidly from limited data, mirroring human intuition? This critical problem is the focus of a recent discussion featuring Ankit Gupta, General Partner at Y Combinator, and Francois Chaubard, Visiting Partner at Y Combinator. They delve into the concept of 'world models' as a potential breakthrough, exploring the motivations, mathematics, and real-world applications that could unlock AGI.

The Sample Efficiency Gap

Gupta frames the core challenge as optimizing for 'intelligence per watt' and 'intelligence per sample.' He highlights that while humans can master new skills with just a handful of tries, current state-of-the-art AI models often need tens of thousands of data points. This inefficiency is a major hurdle, particularly in complex domains like robotics and self-driving cars, where real-world data collection is expensive and time-consuming.

The full discussion can be found on YC's YouTube channel.

The discussion draws a parallel to human learning, where innate genetic encoding and learned experiences contribute to an implicit 'world model' within the brain. This internal model allows for prediction and planning, a capability that current AI models often lack in an explicit form.

World Models: The Core Concept

The central thesis revolves around 'world models', AI systems that aim to build an internal representation of how the world works. These models learn the transition function, essentially predicting the next state (S_t+1) given the current state (S_t) and an action (u_t). This predictive capability is key to improving learning efficiency.

The presenters use the example of Newtonian physics, which serves as a perfect, albeit simplified, world model. This allows for precise predictions of object trajectories without needing to collect millions of real-world data points. This concept is further illustrated with NASA's asteroid interception plans and SpaceX's rocket landing systems, both relying on sophisticated internal models of physics to guide actions.

Human Intuition and World Models

The conversation highlights how humans naturally build and utilize world models. Even mental rehearsal, like imagining shooting a basketball, can lead to significant improvement, demonstrating the power of internal simulation. Neuroscientist Shaw Duckman's theory that the neocortex's expansion was driven by the need for better world modeling underscores this point.

The discussion contrasts the implicit world understanding in natural language models with the explicit need for such models in physical domains like robotics. While LLMs can perform surprisingly intelligent tasks through pattern matching in vast text data, this understanding can break down in scenarios requiring direct interaction with the physical world.

The Mathematics of Optimal Control

The presenters break down the problem of controlling a drone as a practical example. The state vector includes position and velocity, and the goal is to reach a target state. The transition function, governed by physics (F=MA), allows for precise prediction. This leads to the concept of 'model predictive control,' where a model is used to optimize actions over time to minimize a loss function, such as deviation from a target or energy expenditure.

However, the introduction of an adversary, like another drone trying to intercept, transforms the problem into a stochastic and non-differentiable one. This is where traditional optimization breaks down, necessitating approaches like reinforcement learning (RL). The video touches upon various RL techniques like value iteration, policy iteration, DQN, and actor-critic methods, all aimed at modeling these complex, non-differentiable processes.

Challenges and Scaling Issues

The discussion then pivots to the limitations of current approaches, particularly those used in games like Chess and Go. While AlphaGo achieved remarkable success, its scalability is hampered by the combinatorial explosion of states and the computational cost of planning. In real-world applications like self-driving cars and robotics, the state space is effectively infinite, and the need for real-time decision-making is paramount.

The challenge of non-differentiability in complex environments, where the actions of other agents (like other cars on the road) are unknown, further complicates model-based approaches. This forces a reliance on RL, which, while powerful, is often described as 'brutal' and sprawling due to the variety of algorithms and the difficulty in modeling stochastic processes.

The video emphasizes that the success of AlphaGo was contingent on a small action space and a deterministic environment. Real-world scenarios, like the stock market or venture capital, are constantly changing, making a fixed world model insufficient. The need for real-time, adaptive models is clear.

The Path Forward: World Action Models

The conversation highlights the emergence of 'world action models' (WAMs), which jointly model state and action distributions. This approach aims to overcome the computational expense of sampling world models and then passing those actions back into the system. By having a single model output both the action and the next state, WAMs promise greater efficiency.

The discussion concludes by outlining the progression of environments from Chess to Go, then to self-driving cars and robotics, each presenting increasing complexity in state and action spaces. The core takeaway is that world models, by enabling predictive simulation and learning from fewer samples, might be the crucial missing piece in the pursuit of AGI.