The relentless pursuit of scale in Large Language Models (LLMs) has primarily focused on increasing depth. However, this architectural evolution is hampered by a fundamental challenge: signal degradation. As information propagates through numerous residual updates in deeper layers, crucial features learned in shallower layers become diluted, hindering model performance. Addressing this directly, researchers have introduced Mixture-of-Depths Attention (MoDA), a novel mechanism designed to preserve and leverage information across different depths within the model.
Revitalizing Deep Layer Information Flow
MoDA fundamentally redefines how attention heads operate. Instead of solely attending to key-value pairs within the current layer, each attention head is empowered to access sequence KV pairs from its current layer alongside depth KV pairs from preceding layers. This cross-depth attention capability acts as a potent countermeasure against the signal dilution problem. By allowing deeper layers to directly reference and integrate information from shallower layers, MoDA ensures that valuable learned features are not lost but are instead reinforced and utilized more effectively as the model processes information through its increasing depth.
