Current video generation models falter when simulating the physical consequences of 3D actions, a limitation stemming from their lack of structural understanding. This gap prevents them from accurately modeling forces, robotic manipulations, and other physics-driven interactions within 3D scenes.
Bridging Action to Visuals via Physics
The core innovation of RealWonder, the first real-time system for action-conditioned video generation from a single image, lies in its use of physics simulation as an intermediary. Instead of directly encoding complex, continuous actions into visual representations, RealWonder translates these actions through a physics engine. This process generates visual outputs like optical flow and RGB data that existing video models can readily process. This approach fundamentally redefines how we can imbue generated videos with physical realism.
Real-Time Interactive Simulation for Diverse Materials
RealWonder integrates three key components: 3D reconstruction from single images, a physics simulation module, and a highly distilled video generator that requires only four diffusion steps. This architecture enables impressive real-time performance, achieving 13.2 FPS at a resolution of 480x832. The system supports interactive exploration of forces, robot actions, and camera controls across a variety of object types, including rigid bodies, deformable materials, fluids, and granular substances. This capability opens significant avenues for applications in immersive experiences, AR/VR, and robot learning, marking a substantial advancement in RealWonder action conditioned video generation.
