"It's just like the whole thing works, it's kind of mind-blowing." This sentiment, expressed by Karol Hausman of Physical Intelligence, captures the palpable excitement surrounding the potential of foundation models to unlock true general-purpose robotics. Hausman spoke with Alfred Lin and Sonya Huang at Sequoia Capital about their end-to-end learning approach, which seeks to overcome what they identify as the primary bottleneck in robotics: intelligence, not hardware.
The core of Physical Intelligence’s philosophy rests on creating a single, general-purpose model capable of handling diverse physical tasks across different robot embodiments. This contrasts sharply with the traditional, task-specific programming that has historically constrained the field. Instead of painstakingly engineering a solution for every potential action—from pouring coffee to folding laundry—the team is leveraging the scale and generalization power inherent in modern foundation models.
Their work is centered around models like $pi0$ and $pi0.6$, which combine vision, language, and action into a unified learning framework. This integration is crucial for developing robotic intelligence that mirrors human generalization capabilities. As Hausman noted, "If you look at the history of robotics, it's very, very clear to me... that we've always been bottlenecked on intelligence." The ability of these new models to ingest multimodal data—vision, language, and action—is what allows them to begin bridging this intelligence gap.
The team prioritizes real-world deployment, pushing beyond what imitation learning alone can achieve by incorporating Reinforcement Learning from experience. This iterative, self-improving loop is vital for creating robust behaviors that can adapt to novel, unstructured environments. As they discussed, this approach is necessary because relying solely on internet-scale data for robotic actions is insufficient. "There is no data of robots actually operating in the real world," Hausman pointed out, underscoring the need for the robot to learn from its own interactions.
This reliance on self-collected, real-world data collection is where the true scalability challenge lies. The team recognized early on that simply scaling up existing methods wouldn't suffice for generalized physical intelligence. They identified a critical bottleneck: the sheer difficulty of modeling the entire physical world accurately enough for reliable deployment.
The challenge, as they elaborated, is multifaceted. While large language models (LLMs) have shown remarkable proficiency in reasoning over text and abstract spaces, applying that directly to physical control is not straightforward. They found that simply trying to scale up models trained only on internet data—which lacks the necessary grounding in physics and consequence—led to models that were impressive in simulation but brittle in reality. "You can't just write every single case you’ll encounter in the real world," Hausman explained, highlighting the impossibility of enumerating all real-world complexities.
Their solution involves an end-to-end system where the model directly maps sensory inputs (pixels) to actions, bypassing many of the brittle, hand-engineered intermediate layers common in older robotics approaches. This end-to-end paradigm allows for a more fluid, integrated understanding of perception, planning, and control.
The success of this approach is demonstrated by the fact that their models can perform complex, multi-step tasks, such as making coffee for 13 hours straight, without explicit task-specific programming. Furthermore, the generalizability across different robotic embodiments and environments is a key metric of success.
The core difficulty they are tackling is ensuring that when a model learns a task in a controlled environment (like building a cardboard box), that knowledge transfers reliably when deployed in a completely new setting, like a home kitchen. This transferability is where RL from experience becomes paramount, allowing the model to refine its understanding of the physical dynamics through interaction.
"The problem is not how you move your body, it's more like you think through the motion itself," Hausman suggested, indicating a shift from low-level motor control to higher-level, grounded reasoning embedded within the foundation model architecture. This focus on embodied intelligence, where the model understands the physical consequences of its actions, is what Physical Intelligence believes will ultimately bridge the gap between impressive language models and truly useful, general-purpose robots.
