Multi-head attention & masks
Attention can run several times in parallel, and a mask decides which tokens may look at which.
Several heads, several relationships
A single attention map gives each token one weighting over the others. Multi-head attention runs H of these in parallel, each on its own smaller query, key and value projection. One head may follow the previous word, another a verb’s subject. The results are joined and passed through a linear layer.
Masks decide who can see whom
Global attention lets every token read every other token. Causal attention hides later tokens: before the softmax, those scores are set to minus infinity, so their weight becomes zero and each row still sums to 1.
Pick a head and a mask in the lab. Rows are the token asking, columns the token being read.
A worked example
Take a model with d = 512 and 8 heads. Each head projects every token to queries, keys and values of 512 / 8 = 64 numbers, computes its own 64-number attention, and produces a 64-number output per token. The eight outputs are placed side by side, back to 512 numbers, and a final 512 × 512 linear layer mixes them. The cost is about the same as one big head, but the model gets eight different attention patterns.
Encoders, decoders and masks
The mask is just a table of 0s and 1s added before the softmax (as minus infinity where blocked). An encoder such as BERT uses no mask: every token sees the whole sentence. A decoder such as GPT uses the causal lower-triangular mask so that training on a full sentence never lets a position peek at the word it must predict.
Easy to confuse
- Heads are not assigned roles by hand; the patterns such as “previous word” emerge from training, and many heads are hard to interpret.
- The mask changes which tokens can be read, not the size of the matrix: it is still N × N, with the blocked part set to zero weight.
In causal attention, which cells of the attention matrix are blocked?
Row i is the token asking; column j is the token it may read. Blocking j > i means a token never sees words that come after it.
Why use several heads instead of one big one?
One attention map can express one weighting. Multiple heads run in parallel on smaller slices, and their outputs are concatenated and mixed by a linear layer.
Which models use global (unmasked) attention?
An encoder reads the whole input before producing anything, so each token may attend in both directions. A generator must not look ahead.
Read more in the books
- Build a Large Language Model (From Scratch), Sebastian Raschka: Ch. 3, “Coding Attention Mechanisms”: hiding future words with causal attention, and multi-head attention.
- Dive into Deep Learning, Zhang, Lipton, Li, Smola: Ch. 11, “Attention Mechanisms and Transformers”: multi-head attention.
- TinyML and Efficient Deep Learning Computing, Lecture 12: Transformer and LLM (slides), Song Han, MIT 6.5940: section I, multi-head attention and attention masking.