LLM Lab

KV cache, MQA & GQA

Generating text one token at a time means remembering every earlier token. That memory, the KV cache, often costs more than the model itself.

Where this sits in a transformer
Tokenizer→Embedding→Positional encoding→Multi-head attention→Feed-forward→LayerNorm + residual→Linear head→Sampling
Transformer design variants
Encoder / decoder→Positional encoding→KV cache: MHA → MQA → GQA→FFN → GLU

The cache grows with length and batch

To produce token t, attention needs the keys and values of tokens 1 … t − 1. Recomputing them each step would waste work, so they are stored: the KV cache. Its size is batch × layers × kv-heads × head size × length × 2 (keys and values) × bytes per number. Long prompts and many users make it huge.

Share keys and values

The size is proportional to the number of key/value heads. Multi-query attention uses one for all query heads; grouped-query attention uses a few, each shared by a group of queries. Sliding-window attention instead limits how many past tokens each token reads.

Set up a model in the lab and compare the cache with the weights.

A worked example

A 70-billion-parameter Llama 2 has 80 layers and head size 128. With full multi-head attention it would keep 64 key/value heads, so one 4,096-token sequence needs 1 × 80 × 64 × 128 × 4,096 × 2 × 2 bytes ≈ 10.7 GB of cache. Sixteen such sequences need about 170 GB, more than the 140 GB of weights. Llama 2 70B uses grouped-query attention with 8 key/value heads, which divides the cache by 8.

1batch×80layers×64kv-heads×128head size×4096tokens×2K and V×2bytes (fp16)=≈ 10.7 GB
MHA · 64 kv-heads10.7 GB
GQA · 8 kv-heads1.34 GB
MQA · 1 kv-head0.17 GB
KV cache of a 70B Llama 2 for one 4,096-token sequence, as in the lecture. Sharing key/value heads divides the cache: grouped-query attention with 8 kv-heads makes it 8× smaller.

Why it matters

Generation is usually limited by memory, not arithmetic: each new token reads the whole cache. A smaller cache means longer contexts and more users served per GPU. MQA shrinks it most but can cost accuracy; GQA keeps accuracy close to MHA at large sizes, which is why most recent open models use it. Sliding-window attention, as in Mistral 7B, caps the cache at the last W tokens instead.

Easy to confuse

Check yourself · 1 of 3

During generation, what does the KV cache save from being recomputed?

Check yourself · 2 of 3

A model has 64 query heads. With grouped-query attention using 8 key/value heads, how much smaller is the KV cache than with multi-head attention?

Check yourself · 3 of 3

How does the KV cache grow with the number of requests served at once?

Read more in the books

  • Hands-On Large Language Models, Jay Alammar, Maarten Grootendorst: Ch. 3, looking inside large language models (key-value cache, multi-query and grouped-query attention).
  • AI Engineering, Chip Huyen: Ch. 9, “Inference Optimization”.
  • TinyML and Efficient Deep Learning Computing, Lecture 12: Transformer and LLM (slides), Song Han, MIT 6.5940: section II, KV cache optimizations.
All books
Labcache size calculator
How to read this
  • Pick a model and an attention type, then drag the sequence length and batch size. The two tiles compare the KV cache with the model’s weights.
  • The head diagram shows 16 query heads (blue) sharing key/value heads (orange and green) in dashed groups: 16 groups for MHA, 2 for GQA, 1 for MQA.
  • The generation strip shows the cache growing: each step adds one token’s key and value; only the newest token (outlined) computes a query.
Model
Attention
each step adds one key and one value per kv-head to the cache
KV cache–
Weights (fp16)–