The insatiable demand for longer context windows in large language models is hitting a hard ceiling: the memory bandwidth and capacity required for key-value (KV) caches. Existing compression techniques often apply blanket strategies, leading to performance degradation when specific layers or tokens require nuanced preservation. This challenge is directly addressed by the introduction of DepthWeave-KV, a novel token-adaptive cache compression method detailed in recent findings from Anna Cordoba, Adam Puente Tercero, Nerea Angulo Hijo, Mar Linares Tercero, Julia Barrientos et al. on arXiv.
Cross-Depth Factorization for Leaner Caches
DepthWeave-KV revolutionizes KV cache management by factorizing key and value states across adjacent transformer layers. It leverages shared low-rank channel bases, a departure from uniform compression approaches. Crucially, it retains lightweight, token-specific residuals only where attention mechanisms are most sensitive. This cross-depth residual factorization is coupled with a token-conditional depth router. This router intelligently allocates higher reconstruction ranks to tokens that carry instructions or are critical for retrieval, ensuring that semantically rich information is prioritized. Furthermore, the system employs calibration-free online error tracking, utilizing attention-output probes to dynamically adapt compression during generation, eliminating the need for base model retraining.