LLM Lab

Attention

Attention lets every token look at the others and decide which ones matter for its meaning in this sentence.

Where this sits in a transformer
Tokenizer→Embedding→Positional encoding→Multi-head attention→Feed-forward→LayerNorm + residual→Linear head→Sampling

Queries, keys and values

Each token’s embedding is projected into three vectors. Its query asks “what am I looking for?”, every key says “what do I contain?”, and every value is the information to pass on. It works like a video search: the query is what you type, the keys are the video titles, and the values are the videos you get back.

For N tokens with vectors of size d, Q, K and V are N × d matrices. Q·Kᵀ gives an N × N table of scores, one row per token. In the lab they keep the same colours throughout: query, key, value.

Four steps

The score between two tokens is the dot product of one’s query with the other’s key. Dividing by √d keeps the scores in a sensible range, and the causal mask removes later tokens. A softmax turns each row of scores into weights that sum to 1, and the output is the weighted average of the values.

·
=
Shapes: Q (N × d = 5 × 4) times Kᵀ (4 × 5) gives the N × N score matrix, one row per query token. In a decoder, the hatched upper triangle is masked.

Step through the lab with it as the query: it ends up attending mostly to cat, which is how a model works out what a pronoun refers to. Click another token to follow it instead. The vectors are small and hand-made, but the arithmetic is the real thing. Real models run many such heads in parallel and stack dozens of layers.

A worked example

The query of “it” is (3, 0, 0, 0), a vector that looks for animals. The key of “cat” is (3, 0, 0.5, 0), so their dot product is 9. The key of “the” is (0, 0, 0, 1), giving 0. Divided by √d = 2, the scores become 4.5 and 0; after softmax over all earlier tokens, “cat” receives about three quarters of the weight, and the output for “it” is mostly the value of “cat”.

queryit 3000
theq·k = 0÷ 2 = 0.01%
catq·k = 9÷ 2 = 4.579%
satq·k = 1.5÷ 2 = 0.82%
onq·k = 0÷ 2 = 0.01%
itq·k = 6÷ 2 = 3.018%
One row of attention for the query “it”, over five of the tokens before it. Scores are dot products with each key, scaled by √d = 2, then softmaxed: “cat” takes most of the weight.

In a language model

The query, key and value vectors are produced by three learned matrices applied to each token’s vector. Each head works on a slice of 64 to 128 numbers, a model has dozens of heads per layer, and the same computation runs in every layer. Because the score matrix is N × N, the work grows with the square of the context length.

Easy to confuse

Check yourself · 1 of 3

In self-attention, where does a token’s new representation come from?

Check yourself · 2 of 3

What does the causal mask do in a decoder-only model like GPT?

Check yourself · 3 of 3

Why divide the scores by √d before the softmax?

Read more in the books

  • Build a Large Language Model (From Scratch), Sebastian Raschka: Ch. 3, “Coding Attention Mechanisms”.
  • Dive into Deep Learning, Zhang, Lipton, Li, Smola: Ch. 11, “Attention Mechanisms and Transformers”.
  • Hands-On Large Language Models, Jay Alammar, Maarten Grootendorst: Ch. 3, looking inside large language models.
All books
Labstep through Q, K, V
How to read this
  • Click a token to make it the query (outlined). The stage map above the picture shows which of the five steps you are on; the card below explains it.
  • Each row is a token in its own colour. Strips are vectors: blue edge = query, orange = key, green = value; teal cells are positive, orange negative.
  • Step 2 lists q·k for every token, step 3 the scaled scores (later tokens masked), step 4 the weights as bars, and step 5 the weighted values, their sum, and the full N × N matrix (hatched cells are masked).