State2State: Self-Supervised LLM Agent Training

The quest for more capable and adaptable LLM agents often hits a wall: the reliance on externally defined tasks and human supervision. Traditional methods like supervised fine-tuning on expert trajectories or reinforcement learning with handcrafted verifiers, while effective, inherently limit the breadth and scalability of agent training. This bottleneck hinders the development of agents that can truly learn and interact with complex environments autonomously.

Environment-Derived Objectives

Researchers have introduced State2State, a novel environment learning paradigm that shifts the paradigm for training LLM agents. Instead of relying on human-defined goals, State2State converts explored environment states into training objectives. This allows agents to acquire interaction and manipulation capabilities solely through exploration, challenging them to reach specified target states derived directly from their environmental interactions. This method, detailed on arXiv, provides a scalable and verifiable approach to objective generation, sidestepping the need for expert supervision or manual task design.

Scalable and Verifiable Training

The core innovation lies in its ability to generate diverse and verifiable training tasks organically. By deriving objectives from the environment itself and employing rule-based state matching for verification, State2State offers a robust framework for LLM agent environment learning. Experiments on ALFWorld and ScienceWorld demonstrate that this environment learning stage significantly improves agent performance as a standalone component. Furthermore, when used as an initialization for downstream reinforcement learning, State2State boosts final performance and accelerates learning efficiency, with promising evidence of its ability to generalize across different environments.