Reinforcement Learning from AI Feedback (RLAIF) has emerged as a powerful paradigm for improving language models, yet the underlying mechanisms of its self-improvement in value learning remain theoretically underexplored. This work proposes the latent value hypothesis, offering a novel explanation for why RLAIF effectively aligns models with desired behaviors.
Human Values as Latent Representation Directions
The core insight is that extensive pretraining on internet-scale data imbues language models with latent representations of human values. These values are not explicitly programmed but exist as discernible directions within the model's representation space. Constitutional prompts, when applied, act as projection operators, selectively eliciting these latent value-aligned directions into observable preference judgments. This formalization, as detailed in a paper by Robin Young on arXiv, provides a theoretical framework for understanding RLAIF explained.
The Generation-Judgment Gap and Alignment Ceilings
The latent value hypothesis explains the observed generation-judgment gap in RLAIF. Alignment improves when the direction activated by the constitution correlates better with true values than the model's default generation direction. Crucially, the theoretical ceiling on RLAIF quality is directly tied to how well the model's representations encode these values, a capability that scales with model capacity. This suggests that larger, more capable models inherently possess a greater potential for value alignment through RLAIF.
Adversarial Constitutions and Unifying Empirical Findings
The framework also illuminates potential failure modes. Adversarial constitutions can be constructed to activate anti-social value directions, inadvertently encoded from harmful pretraining data. This account unifies scattered empirical observations, including the emergence of refusal directions in models, the existence of low-rank safety subspaces, and the observed scaling behavior of RLAIF. Understanding RLAIF explained through this lens is critical for developing robust and safe AI systems.
