AI Research

Unlocking Ultra-Long Context for LLMs

MiniMax Sparse Attention breaks the context window barrier for LLMs, enabling millions of tokens with significant compute reduction and practical speedups.

StartupHub.ai Staff

By StartupHub.ai Staff, reviewed by Daniel Singer, Editor

2 min read

Diagram illustrating the MiniMax Sparse Attention mechanism
Conceptual overview of the MiniMax Sparse Attention mechanism.
Visual TL;DR
LLM context limitDriver
quadratic cost of standard softmax attention hinders ultra-long context
Demand for long contextDriver
agentic workflows, code reasoning, persistent memory require millions of tokens
From the articleThe insatiable demand for ultra-long context capabilities in frontier LLMs, spanning agentic workflows, repository-scale code reasoning, and persistent memory, is currently stymied by the quadratic cost of standard softmax attention.
MiniMax Sparse AttentionCore
From the article 2 mentionsTo surmount this challenge, the researchers introduce MiniMax Sparse Attention (MSA), a novel blockwise sparse attention mechanism built upon Grouped Query Attention (GQA).
Index BranchCore
From the article 2 mentionsMSA employs a lightweight Index Branch to score and select a Top-k subset of key-value blocks for each GQA group, enabling group-specific sparse retrieval.
Main BranchCore
From the article 2 mentionsThe Main Branch then executes exact block-sparse attention exclusively over these selected blocks.
Breaks context barrierEffect
enables millions of tokens with significant compute reduction
From the articleThis computational barrier renders models untenable at deployment scale for contexts stretching into the millions of tokens.
Practical speedupsOutcome
optimized for GPU execution and efficient deployment across architectures
From the articleCrucially, when paired with its optimized kernel, MSA delivers substantial wall-clock speedups: 14.2x for prefill and 7.6x for decoding on H800 hardware.

The insatiable demand for ultra-long context capabilities in frontier LLMs, spanning agentic workflows, repository-scale code reasoning, and persistent memory, is currently stymied by the quadratic cost of standard softmax attention. This computational barrier renders models untenable at deployment scale for contexts stretching into the millions of tokens.

Block-wise Sparsity Meets Grouped Query Attention

To surmount this challenge, the researchers introduce MiniMax Sparse Attention (MSA), a novel blockwise sparse attention mechanism built upon Grouped Query Attention (GQA). MSA employs a lightweight Index Branch to score and select a Top-k subset of key-value blocks for each GQA group, enabling group-specific sparse retrieval. The Main Branch then executes exact block-sparse attention exclusively over these selected blocks. This design prioritizes simplicity and scalability, facilitating efficient deployment across diverse GPU architectures.

Optimized for GPU Execution and Practical Speedups

Translating theoretical sparsity into tangible performance gains required a co-designed GPU execution path. MSA leverages exp-free Top-k selection and KV-outer sparse attention to enhance tensor-core utilization under block-granular access. On a 109B-parameter multimodal model, MSA achieves performance parity with GQA while slashing per-token attention compute by an impressive 28.4x at a 1 million token context. Crucially, when paired with its optimized kernel, MSA delivers substantial wall-clock speedups: 14.2x for prefill and 7.6x for decoding on H800 hardware.

Sources

Get the free daily brief every morning

Every morning the day's AI news, new AI startups and new funding rounds. Free and takes a few minutes to read, unsubscribe at any time.

Join 64,000 other subscribers.

TopicsAI ResearchLLM OptimizationSparse AttentionLong Context

StartupHub.ai Staff

Written by

StartupHub.ai StaffEditorial team

The staff writers of StartupHub.ai, ranging from investment analysts to avid AI tool users, early adopters and critical enthusiasts. Backgrounds span engineering, business and the arts. We hold every piece to rigorous standards of research and review.

Daniel Singer

Reviewed by

Daniel Singer is an AI expert who has been writing about, covering and building AI technology since 2018.

Test your Agent Readiness now

Paste your website. How well ChatGPT, Claude and the other agents can read and use a site, in about a minute.

Watch

The videos behind the news, one flip at a time

The feed mixes the videos from our articles with startup demos and you flip through them like TV channels for free with no signup.