The insatiable demand for longer context windows in large language models is hitting a hard ceiling: the memory bandwidth and capacity required for key-value (KV) caches. Existing compression techniques often apply blanket strategies, leading to performance degradation when specific layers or tokens require nuanced preservation. This challenge is directly addressed by the introduction of DepthWeave-KV, a novel token-adaptive cache compression method detailed in recent findings from Anna Cordoba, Adam Puente Tercero, Nerea Angulo Hijo, Mar Linares Tercero, Julia Barrientos et al. on arXiv.
Cross-Depth Factorization for Leaner Caches
DepthWeave-KV revolutionizes KV cache management by factorizing key and value states across adjacent transformer layers. It leverages shared low-rank channel bases, a departure from uniform compression approaches. Crucially, it retains lightweight, token-specific residuals only where attention mechanisms are most sensitive. This cross-depth residual factorization is coupled with a token-conditional depth router. This router intelligently allocates higher reconstruction ranks to tokens that carry instructions or are critical for retrieval, ensuring that semantically rich information is prioritized. Furthermore, the system employs calibration-free online error tracking, utilizing attention-output probes to dynamically adapt compression during generation, eliminating the need for base model retraining.
Accelerated Inference with Fused Implementation
Beyond the architectural innovation, the practical deployment of DepthWeave-KV is enhanced by a fused CUDA implementation. This optimization seamlessly integrates basis lookup, residual dequantization, and attention projection. The result is a significant reduction in decode-time memory traffic. Tested across demanding benchmarks like LongBench, Needle-in-a-Haystack, and L-Eval, alongside long-form QA and summarization tasks, DepthWeave-KV demonstrates its efficacy. The method achieves near-full-cache task quality while drastically cutting memory usage, outperforming prior compressed caches in both average score and retrieval accuracy. This translates to a remarkable 8.3x KV memory reduction and a throughput of 72.8 tokens per second at a 64K context length.