KV cache, MQA & GQA
Generating text one token at a time means remembering every earlier token. That memory, the KV cache, often costs more than the model itself.
The cache grows with length and batch
To produce token t, attention needs the keys and values of tokens 1 … t − 1. Recomputing them each step would waste work, so they are stored: the KV cache. Its size is batch × layers × kv-heads × head size × length × 2 (keys and values) × bytes per number. Long prompts and many users make it huge.
Share keys and values
The size is proportional to the number of key/value heads. Multi-query attention uses one for all query heads; grouped-query attention uses a few, each shared by a group of queries. Sliding-window attention instead limits how many past tokens each token reads.
Set up a model in the lab and compare the cache with the weights.
A worked example
A 70-billion-parameter Llama 2 has 80 layers and head size 128. With full multi-head attention it would keep 64 key/value heads, so one 4,096-token sequence needs 1 × 80 × 64 × 128 × 4,096 × 2 × 2 bytes ≈ 10.7 GB of cache. Sixteen such sequences need about 170 GB, more than the 140 GB of weights. Llama 2 70B uses grouped-query attention with 8 key/value heads, which divides the cache by 8.
Why it matters
Generation is usually limited by memory, not arithmetic: each new token reads the whole cache. A smaller cache means longer contexts and more users served per GPU. MQA shrinks it most but can cost accuracy; GQA keeps accuracy close to MHA at large sizes, which is why most recent open models use it. Sliding-window attention, as in Mistral 7B, caps the cache at the last W tokens instead.
Easy to confuse
- MQA and GQA reduce key/value heads only; the number of query heads, and the model’s width, stay the same.
- The KV cache is per sequence and grows with every token; the weights are shared by all sequences and stay fixed.
During generation, what does the KV cache save from being recomputed?
Each new token needs the keys and values of all earlier tokens. Storing them means a step computes only the new token’s query, key and value.
A model has 64 query heads. With grouped-query attention using 8 key/value heads, how much smaller is the KV cache than with multi-head attention?
The cache stores one key and one value per kv-head. Going from 64 kv-heads to 8 divides its size by 8; multi-query attention (1 kv-head) would divide it by 64.
How does the KV cache grow with the number of requests served at once?
Every sequence has its own keys and values, so serving more sequences at once multiplies the cache.
Read more in the books
- Hands-On Large Language Models, Jay Alammar, Maarten Grootendorst: Ch. 3, looking inside large language models (key-value cache, multi-query and grouped-query attention).
- AI Engineering, Chip Huyen: Ch. 9, “Inference Optimization”.
- TinyML and Efficient Deep Learning Computing, Lecture 12: Transformer and LLM (slides), Song Han, MIT 6.5940: section II, KV cache optimizations.