LOCKS Unlocks LLM Long Context Efficiency

Serving large language models with extensive context windows has been fundamentally bottlenecked by the memory and computational demands of the Key-Value (KV) cache. This cache requires full reads at every decoding step, creating a significant performance impediment. The LOCKS framework, detailed on arXiv, introduces a paradigm shift by recognizing that attention keys, while globally high-rank, exhibit local low-rank properties.

Spectral Summaries for Efficient Attention Selection

LOCKS tackles this challenge by providing each KV cache 'page' with its own spectral summary. This summary, approximately one-tenth the size of the full cache, allows the system to reconstruct within-page logits and estimate each page's attention mass using a log-sum-exp approach. Crucially, this enables LOCKS to attend only to the top-ranked pages, drastically reducing the number of keys and values that need to be read without processing candidate keys. This approach maintains performance, staying within a point of full-cache quality on long-document QA benchmarks like LongBench-v1 and closely tracking an oracle on retrieval-dense tasks like RULER, even at the smallest computational budgets.

Near Full-Quality Inference at Extreme Context Lengths

The efficacy of the LOCKS KV cache optimization becomes particularly pronounced in demanding long-form reasoning tasks, where baseline selectors often fail. At a practical $2048$-token budget, LOCKS achieves aggregate quality comparable to a full KV cache with over $100$K$+$ context, while only attending to about $2%$ of the tokens. This translates to a halving of per-token decode latency (a $2.0 imes$ improvement at $1$M$ tokens) compared to dense attention mechanisms. The system is engineered as a drop-in plugin for unmodified vLLM, leveraging CUDA graphs for batched decoding and offering a seamless path to deployment.