KV cache & quantisation
Generating text one token at a time would be slow if the model reread the whole prefix every step. The KV cache trades memory for that time.
Why cache
Each new token attends to every earlier token, which needs their keys and values in every layer. Those don’t change once computed, so the server keeps them: the KV cache.
Its size is 2 × layers × kv_heads × head_dim × tokens × batch × bytes. The 2 is for keys and values.
Shrinking it
Grouped-query attention shares key/value heads across query heads. Quantisation stores numbers in fewer bits: 16-bit to 8-bit halves memory, to 4-bit quarters it, at some cost in accuracy. Long contexts and large batches are often limited by this cache rather than by the weights.
To see why the cache grows with every generated token, and how multi-query and grouped-query attention shrink it, step through KV cache, MQA & GQA.
A worked example
A 7-billion-parameter Llama 2 has 32 layers, 32 heads and head size 128. In fp16 its cache needs 32 × 32 × 128 × 2 (keys and values) × 2 bytes = 512 KB per token, so a 4,096-token context takes 2 GB per sequence, and 16 sequences take 32 GB, more than the 13.5 GB of weights. Storing the cache in 8 bits halves it; 4 bits quarters it.
Quantisation
Quantisation stores numbers on a coarse grid of 2^bits levels. Weights quantised to 8 bits usually lose almost no quality, and 4-bit weights are common for running models on a single GPU or a laptop. Outliers, a few very large values, are the main difficulty: they stretch the grid and waste levels, which is what methods like SmoothQuant address.
Easy to confuse
- Weights and the KV cache are quantised separately; one can be 4-bit while the other stays 16-bit.
- Fewer bits reduce memory and bandwidth, which is usually what limits generation speed, more than they reduce arithmetic.
What does the KV cache store?
When generating token t, attention needs the keys and values of tokens 1..t−1. Caching them avoids recomputing the whole prefix every step.
Doubling the context length does what to the KV cache?
The cache holds one key and one value per token per layer, so it grows linearly with sequence length (and with batch size).
Grouped-query attention cuts the cache by…
With 8 KV heads serving 32 query heads, the cache is 4× smaller while the number of query heads stays the same.
Read more in the books
- AI Engineering, Chip Huyen: Ch. 9, “Inference Optimization”.
- Hands-On Large Language Models, Jay Alammar, Maarten Grootendorst: Ch. 3, looking inside large language models (caching keys and values).