Microsoft Research has unveiled AsgardBench, a new benchmark designed to rigorously test the ability of AI agents to plan and adapt tasks based on visual input. This development addresses a critical gap in evaluating embodied AI, which requires agents to interact with and understand their environment.
Unlike previous benchmarks that often bundle perception, navigation, and control, AsgardBench isolates the crucial aspect of visually grounded interactive planning. It challenges AI agents to adjust their actions in simulated household tasks when visual observations contradict their initial assumptions. This is vital for creating robots and AI systems capable of navigating the unpredictable real world.
The benchmark, detailed in the paper "AsgardBench, Evaluating Visually Grounded Interactive Planning Under Minimal Feedback," presents AI agents with 108 controlled task instances across 12 categories. The core idea is simple: give an agent a goal, let it see the environment, and observe if it can revise its plan when reality doesn't match expectations. For example, an agent tasked with cleaning a mug might find it already clean or the sink already full, requiring a plan modification.
Focus on Adaptation, Not Just Execution
Built upon the AI2-THOR simulation environment, AsgardBench provides agents with a limited action set (find, pickup, put, clean, toggle_on/off) and visual input. At each step, the agent proposes a full plan, but only the first action is executed. This forces continuous re-evaluation and adaptation, moving beyond static, pre-scripted behaviors. The focus is squarely on AI agent plan adaptation, not on basic navigation or object manipulation.
This approach means that identical instructions can necessitate different action sequences depending on the observed state of objects and the environment. The benchmark emphasizes the agent's capacity to notice subtle visual cues, like whether a mug is dirty or a faucet is running, and to maintain task context across multiple steps.