Attention
Attention lets every token look at the others and decide which ones matter for its meaning in this sentence.
Queries, keys and values
Each token’s embedding is projected into three vectors. Its query asks “what am I looking for?”, every key says “what do I contain?”, and every value is the information to pass on. It works like a video search: the query is what you type, the keys are the video titles, and the values are the videos you get back.
For N tokens with vectors of size d, Q, K and V are N × d matrices. Q·Kᵀ gives an N × N table of scores, one row per token. In the lab they keep the same colours throughout: query, key, value.
Four steps
The score between two tokens is the dot product of one’s query with the other’s key. Dividing by √d keeps the scores in a sensible range, and the causal mask removes later tokens. A softmax turns each row of scores into weights that sum to 1, and the output is the weighted average of the values.
Step through the lab with it as the query: it ends up attending mostly to cat, which is how a model works out what a pronoun refers to. Click another token to follow it instead. The vectors are small and hand-made, but the arithmetic is the real thing. Real models run many such heads in parallel and stack dozens of layers.
A worked example
The query of “it” is (3, 0, 0, 0), a vector that looks for animals. The key of “cat” is (3, 0, 0.5, 0), so their dot product is 9. The key of “the” is (0, 0, 0, 1), giving 0. Divided by √d = 2, the scores become 4.5 and 0; after softmax over all earlier tokens, “cat” receives about three quarters of the weight, and the output for “it” is mostly the value of “cat”.
In a language model
The query, key and value vectors are produced by three learned matrices applied to each token’s vector. Each head works on a slice of 64 to 128 numbers, a model has dozens of heads per layer, and the same computation runs in every layer. Because the score matrix is N × N, the work grows with the square of the context length.
Easy to confuse
- Q, K and V are not three different inputs: all three come from the same token vectors through different weights.
- Each row of the attention matrix sums to 1, but the columns do not: one token can be attended to by many others.
In self-attention, where does a token’s new representation come from?
Each token mixes in information from other tokens, weighted by how well its query matches their keys.
What does the causal mask do in a decoder-only model like GPT?
During generation the future doesn’t exist yet, so training masks it out too: each position only sees itself and earlier positions.
Why divide the scores by √d before the softmax?
Dot products of d-dimensional vectors grow with d; large scores push softmax to one-hot and make gradients tiny.
Read more in the books
- Build a Large Language Model (From Scratch), Sebastian Raschka: Ch. 3, “Coding Attention Mechanisms”.
- Dive into Deep Learning, Zhang, Lipton, Li, Smola: Ch. 11, “Attention Mechanisms and Transformers”.
- Hands-On Large Language Models, Jay Alammar, Maarten Grootendorst: Ch. 3, looking inside large language models.