#Attention mechanisms

Unlocking Transformer Potential Beyond Semantics

Researchers propose SIREN-RoPE, unlocking a novel 'rotation space' in Transformers for dynamic relational encoding, yielding consistent performance gains with minimal overhead.

MoDA: Unlocking LLM Depth Scaling

Mixture-of-Depths Attention (MoDA) tackles LLM signal degradation by enabling cross-layer attention, boosting performance with minimal overhead.

Kimi Linear promises to beat full attention with less memory

The Dynamic AI Frontier: Hot-Swapping Skills in Large Language Models