Semantic Quantization in KV Cache Eviction

When a program runs out of memory, the operating system does not crash, it uses a page replacement policy to swap pages in and out, keeping the most useful ones active.

LLMs processing long texts face a similar memory bottleneck. They store a "KV cache" (memory buffer) that grows linearly with sequence length. At 32,000 tokens, this cache requires ~11.6 GB of VRAM. If you try to run this on a standard 8 GB GPU, the process runs out of memory and crashes.

Popular open-source solutions like H2O and SnapKV try to evict less important tokens using attention weights. However, calculating these weights across long contexts creates an even larger memory bottleneck, causing OOM crashes anyway.

I built EntropyKV to solve this using systems design principles rather than complex ML overhead. EntropyKV operates like an OS page replacement policy for model memory:

1. Value-Weighted Key Norms: Instead of computing heavy attention matrices, it decides which tokens to evict using a lightweight metric (the L2 norms of key and value vectors). This is highly efficient and compatible with FlashAttention.
2. Layer-Adaptive Allocation: It allocates bigger recency windows to early and late layers (which handle structure and reasoning) while aggressively compressing the middle layers.

Result: EntropyKV cuts cache memory by 43% (11.6 GB → 6.6 GB) and speeds up generation by 32×, allowing long-context models to run on standard 8 GB consumer GPUs where competing methods crash.

Crucially, when cache space is cut by 50%, the model doesn't output garbage. It exhibits what we call "semantic quantization" : it perfectly preserves the structure and meaning of the retrieved context while only losing fine numeric details (e.g., retrieving "2023" instead of "2026").

1. Understand why traditional attention-based KV cache eviction methods fail (OOM) at long contexts on consumer GPUs.
2. Learn how to implement a lightweight, attention-free eviction policy using only key and value vector norms.
3. Explore how layer-adaptive allocation (LARA) acts as a tiered caching strategy across different layers of a model.
4. Witness the "semantic quantization" effect, showing how neural networks gracefully degrade like JPEGs under lossy memory compression.