LLM Lab

KV cache & quantisation

Generating text one token at a time would be slow if the model reread the whole prefix every step. The KV cache trades memory for that time.

Where this sits in the life of an LLM
Pretraining at scale→Supervised fine-tuning→Preference tuning (RLHF, DPO)→Serving (KV cache, quantisation)
Transformer design variants
Encoder / decoder→Positional encoding→KV cache: MHA → MQA → GQA→FFN → GLU

Why cache

Each new token attends to every earlier token, which needs their keys and values in every layer. Those don’t change once computed, so the server keeps them: the KV cache.

Its size is 2 × layers × kv_heads × head_dim × tokens × batch × bytes. The 2 is for keys and values.

Shrinking it

Grouped-query attention shares key/value heads across query heads. Quantisation stores numbers in fewer bits: 16-bit to 8-bit halves memory, to 4-bit quarters it, at some cost in accuracy. Long contexts and large batches are often limited by this cache rather than by the weights.

To see why the cache grows with every generated token, and how multi-query and grouped-query attention shrink it, step through KV cache, MQA & GQA.

A worked example

A 7-billion-parameter Llama 2 has 32 layers, 32 heads and head size 128. In fp16 its cache needs 32 × 32 × 128 × 2 (keys and values) × 2 bytes = 512 KB per token, so a 4,096-token context takes 2 GB per sequence, and 16 sequences take 32 GB, more than the 13.5 GB of weights. Storing the cache in 8 bits halves it; 4 bits quarters it.

32layers×32heads×128head size×2K and V×2bytes=512 KB per token
KV cache per token for a 7-billion-parameter Llama 2 in fp16. A 4,096-token context therefore needs 2 GB per sequence; quantising the cache to 8 bits halves it.

Quantisation

Quantisation stores numbers on a coarse grid of 2^bits levels. Weights quantised to 8 bits usually lose almost no quality, and 4-bit weights are common for running models on a single GPU or a laptop. Outliers, a few very large values, are the main difficulty: they stretch the grid and waste levels, which is what methods like SmoothQuant address.

Easy to confuse

Check yourself · 1 of 3

What does the KV cache store?

Check yourself · 2 of 3

Doubling the context length does what to the KV cache?

Check yourself · 3 of 3

Grouped-query attention cuts the cache by…

Read more in the books

  • AI Engineering, Chip Huyen: Ch. 9, “Inference Optimization”.
  • Hands-On Large Language Models, Jay Alammar, Maarten Grootendorst: Ch. 3, looking inside large language models (caching keys and values).
All books
Labmemory calculator
How to read this
  • Pick a preset or type a model’s layers, heads and head size, choose the context length and the number format; the result is the KV cache per token and per sequence.
  • Below: a row of ten weights in fp16, the same weights rounded to the chosen number of bits, and the rounding error. Fewer bits save memory and add error.
Precision