Visual TL;DR
LLM context limitDriver
quadratic cost of standard softmax attention hinders ultra-long context
Demand for long contextDriver
agentic workflows, code reasoning, persistent memory require millions of tokens
From the articleThe insatiable demand for ultra-long context capabilities in frontier LLMs, spanning agentic workflows, repository-scale code reasoning, and persistent memory, is currently stymied by the quadratic cost of standard softmax attention.
MiniMax Sparse AttentionCore
From the article 2 mentionsTo surmount this challenge, the researchers introduce MiniMax Sparse Attention (MSA), a novel blockwise sparse attention mechanism built upon Grouped Query Attention (GQA).
Index BranchCore
From the article 2 mentionsMSA employs a lightweight Index Branch to score and select a Top-k subset of key-value blocks for each GQA group, enabling group-specific sparse retrieval.
Main BranchCore
From the article 2 mentionsThe Main Branch then executes exact block-sparse attention exclusively over these selected blocks.
Breaks context barrierEffect
enables millions of tokens with significant compute reduction
From the articleThis computational barrier renders models untenable at deployment scale for contexts stretching into the millions of tokens.
Practical speedupsOutcome
optimized for GPU execution and efficient deployment across architectures
From the articleCrucially, when paired with its optimized kernel, MSA delivers substantial wall-clock speedups: 14.2x for prefill and 7.6x for decoding on H800 hardware.