Curiosity · Concept

LLM KV-Cache Compression

During autoregressive decoding, LLMs cache the key and value vectors of every previous token so attention doesn't recompute them. At long context lengths this KV cache dominates GPU memory — often larger than the weights themselves. KV-cache compression covers four main levers: quantization (INT8/INT4 per-head scales), eviction (discard old or low-attention tokens, e.g., StreamingLLM, H2O), low-rank decomposition (DeepSeek's MLA), and token pruning/merging. In production these techniques are stacked to unlock 100k-plus token contexts cost-effectively.

Quick reference

Proficiency
Advanced
Also known as
KV cache quantization, KV cache eviction
Prerequisites
KV cache, Attention mechanism, Quantization

Frequently asked questions

Why is KV-cache compression important?

At 32k+ token contexts, the KV cache can be several times larger than the model weights. Shrinking it is the single biggest lever for long-context and high-batch inference cost.

What are the main KV-cache compression techniques?

Quantization (store K/V in INT8 or INT4), eviction (drop old / low-attention tokens — StreamingLLM, H2O, Scissorhands), low-rank compression (DeepSeek's MLA), and token merging / pruning.

Does KV-cache compression hurt quality?

INT8 KV quantization is near-lossless. INT4 and aggressive eviction can degrade quality on tasks that need distant context. Method choice depends on task: short-horizon chat tolerates more compression than long-document QA.

How does MLA fit into this?

DeepSeek's Multi-Latent Attention is essentially a learned low-rank compression of K/V: only a small latent vector per token is cached, and the model is trained end-to-end to work with it, giving near-MHA quality.

Sources

  1. Xiao et al. — Efficient Streaming Language Models with Attention Sinks (StreamingLLM) — accessed 2026-04-20
  2. Zhang et al. — H2O: Heavy-Hitter Oracle for Efficient Generative Inference of LLMs — accessed 2026-04-20

Related

  • Grouped-Query Attention (GQA) — Grouped-Query Attention (GQA) is an attention variant where multiple query heads share a single key/value head — it cuts KV-cache memory and boosts inference throughput with almost no quality loss versus full multi-head attention.
  • KV Cache — The KV cache stores the key and value tensors computed by self-attention for past tokens so that generating each new token becomes O(1) in sequence length instead of re-processing the entire prefix.
  • INT4 Quantization — INT4 quantization compresses LLM weights from 16-bit floating point down to 4-bit integers — it cuts model memory by ~4x and typically doubles inference throughput with only small quality degradation when paired with modern algorithms like GPTQ or AWQ.
  • All Concepts — Browse the complete concepts library
  • VSET — Programmes offered — B.Tech programs (CSE, AI&ML, AI&DS, Electronics, IIoT)