The quadratic complexity of self-attention has long been a bottleneck for scaling transformer models to handle increasingly long sequences. This limitation hinders progress in domains demanding extensive contextual understanding. A new mechanism, the Polynomial Mixer (PoM), emerges as a direct, linear-complexity alternative.
Polynomial Mixer: A Learned Polynomial Approximation
The Polynomial Mixer (PoM) introduces a paradigm shift by aggregating input tokens into a compressed representation via a learned polynomial function. This compact representation then allows each token to retrieve contextual information. Crucially, the researchers demonstrate that PoM satisfies the contextual mapping property, mathematically ensuring that transformers augmented with PoM retain their capacity as universal sequence-to-sequence approximators. This theoretical grounding is vital for trust and adoption.
Broad Domain Efficacy with Drastic Cost Reduction
The practical implications of PoM are profound. The authors validated its effectiveness by substituting standard self-attention with PoM across five distinct domains: text generation, handwritten text recognition, image generation, 3D modeling, and Earth observation. In each case, PoM achieved performance parity with established attention-based models. The true breakthrough, however, lies in its computational efficiency. When processing long sequences, PoM offers a drastic reduction in computational cost, unlocking scalability previously constrained by attention's quadratic scaling. The availability of the code further accelerates its integration and experimentation.
