Gaurav Mishra: RL Agents Need 'Flight School', Not Just Exams

Gaurav Mishra, a researcher at Amazon AGI Lab, delivered a compelling talk titled "RL to IRL: What breaks when agents trained with reinforcement learning are deployed in real life." Mishra, who previously spent over a decade at Google, including time at DeepMind, highlighted the critical differences and challenges in moving AI agents from simulated environments to the messy reality of computer use.

Understanding Reinforcement Learning and Its Limits

Mishra began with a swift overview of reinforcement learning (RL), defining the agent as the policy that learns through sampling generations and receiving rewards. He contrasted RL with supervised fine-tuning (SFT), noting RL's effectiveness in domains where demonstration data is scarce but tasks are plentiful and have multiple verifiable solutions, such as coding. He emphasized that RL is particularly suited for reasoning-heavy tasks where judging based on outcomes is more effective than on specific token predictions.

The key components of RL, according to Mishra, are the task (which must have a verifiable outcome, be targeted, and within the right difficulty window), the environment (requiring safe sandboxes for execution), and the verifier (which provides the training signal through various judging mechanisms like compilers, linters, or even other agents). He noted that while RL has yielded impressive coding agents, its application to broader computer-use tasks is where the real challenges emerge.

The "Real Life" Problem: When the Game Fights Back

Mishra illustrated the difficulties with real-world deployment through two stark video demonstrations. In the first, an agent tasked with submitting an expense report failed when it encountered a login screen with expired credentials. The agent attempted to guess the password repeatedly, ultimately locking the account. The second demo showed an agent misidentifying an advertisement's submit button as the correct one, leading it to navigate to a different website and begin filling in personal details.

These failures, Mishra explained, stem from the inherent complexities of real-world computer use, which he summarized as several key challenges:

• Partial Observability: The agent's view (screenshots, DOM) is often incomplete, missing dynamically generated content or information requiring scrolling.
• Irreversibility: Actions like form submission or file deletion are often irreversible, making errors costly.
• Non-determinism: Real-world actions are not always predictable due to factors like latency, network issues, or system restarts.
• Ephemeral Authority: Sessions and credentials expire frequently, requiring agents to handle these edge cases gracefully.
• Ambiguous Success: A task might be marked as 'done' but not achieve the desired outcome, such as filing an expense report while also sending a resignation letter.
• Adversarial Content: The digital world is filled with deceptive elements like ads and scams designed to capture attention and mislead.

Mishra pithily stated, "RL worked when the world was a game. IRL starts when the game fights back."

Adapting for Real-World Performance: Flight School, Not Just Exams

To address these challenges, Mishra proposed a shift in training methodology, likening it to needing "flight school, not just exams." This involves simulating the messiness and edge cases of the real world during training. The core components for this adaptation include:

• High-Fidelity Flight Simulators: Training environments must incorporate realistic elements like layout shifts, slow loads, missing labels, pop-ups, and random account states. Crucially, recovery from errors must be a native model action, not just a reset.
• Process Reward Models: Rewards should not only consider the final outcome but also penalize dangerous or unproductive actions taken throughout the process.
• Calibrated Confidence: Agents must learn to assess the risk of their actions and know when to escalate to a human user.
• Adversarial Tasks: Training must actively incorporate scenarios designed to expose and correct potential failures.

Upgrading the Pilot and Cockpit

Mishra elaborated on the necessary upgrades for both the AI model (the "pilot") and its supporting system (the "cockpit").

For the pilot, coding abilities alone are insufficient. Agents need enhanced perception primitives, including:

• Screen Grounding: Understanding the context, visibility, occlusion, and enablement of elements on screen.
• Semantic UI Understanding: Interpreting the purpose of UI elements, such as recognizing a submit button that is actually part of an advertisement.
• Change Detection: Identifying significant changes after an action to understand if the page navigated correctly, if an upload completed, or if a spinner disappeared.
• Multi-Source Observation: Integrating information from screenshots, DOM, browser events, clipboard, and network status to form a comprehensive understanding.

For the cockpit, the "harness" that interfaces the model with the world needs several key features:

• Checkpointing and Rollback: Saving states before risky actions for potential recovery.
• Action Risk Classifier: Evaluating the safety of proposed actions.
• Credential Guardrails: Detecting issues like expired logins and guiding the model.
• Execution Monitors: Identifying unproductive loops or repeated actions.
• Audit Logs: Maintaining a record of all actions and their effects.
• Human Handoff: Empowering the system to transfer control to a user when the agent's confidence is low.

Summary of RL vs. IRL Realities

Mishra concluded by summarizing the shift in assumptions from traditional RL to real-world applications:

• State Observability: RL assumes observable states; reality presents partial and messy UI states, requiring perception primitives.
• Action Cost: RL assumes cheap actions; reality shows actions can be irreversible, necessitating risk-aware execution.
• Reward Clarity: RL assumes clear rewards; reality offers ambiguous success, demanding verification and recovery policies.
• Failure Reset: RL assumes failure resets; reality shows persistent failures, requiring robust recovery strategies.
• Environment Passivity: RL assumes passive environments; reality presents adversarial content, necessitating trust boundaries.
• Autonomy: RL assumes autonomy is always good; reality shows handoff can be optimal, requiring calibrated agency.

He ended with a powerful statement: "The difference between a demo and a product is what happens after the first failed click." This underscores the importance of building resilient, adaptable agents through comprehensive, real-world-informed training.