Bellman Policy Optimization, or BPO, offers a new way to train large language models for reasoning with verifiable rewards without learning a separate critic. Derived from Policy Mirror Descent, the method uses the Bellman equations to rewrite the update as a trajectory-level objective for autoregressive generation with terminal rewards. The reformulation avoids estimating state values at intermediate states, yet the authors prove it shares the same unique optimal solution as the original mirror descent formulation.
That matters because reinforcement learning with verifiable rewards has become the workhorse for improving LLM reasoning, but critic-based or value-estimation steps add variance, memory cost and tuning complexity. By staying critic-free and operating at the trajectory level, BPO promises simpler, more stable optimization for the sparse, outcome-only rewards common in math, code and tool-use tasks, while retaining principled convergence guarantees.
From trajectory rewards to precise credit
The same tension appears in Granularity-Adaptive Credit Assignment for Long-Horizon LLM Agent Reinforcement Learning. That paper notes that group-relative methods like GRPO broadcast one trajectory scalar to every step and cannot say which decision drove the outcome. Its predecessor GiGPO tries to recover step-level signal by grouping time steps that share an anchor state, but still merges step and episode estimates. Together with BPO, the two papers point to a shift in agent RL: keep the simplicity of trajectory-level learning, but restore principled, fine-grained credit without paying for a full critic.
When representations need fixing
Sharp Rates and a One-Line Correction for Spectral Representation Learning adds the representation-learning complement. The work analyzes the self-supervised encoders that are trained once, frozen and reused via lightweight probes, showing that canonical correlation analysis, HGR maximal correlation and the population optimum of the spectral contrastive loss all recover the top-k singular subspace of a cross-view dependence operator. The authors characterize when those off-the-shelf features are good enough under an isotropic task prior and when they need fixing, providing sharp rates and a one-line correction. For teams shipping models, the message converges: training is moving toward objectives that are cheaper to optimize and easier to diagnose, whether by removing the critic, sharpening credit, or correcting the feature subspace directly.
