LLM Lab

Multi-head attention & masks

Attention can run several times in parallel, and a mask decides which tokens may look at which.

Where this sits in a transformer
Tokenizer→Embedding→Positional encoding→Multi-head attention→Feed-forward→LayerNorm + residual→Linear head→Sampling

Several heads, several relationships

A single attention map gives each token one weighting over the others. Multi-head attention runs H of these in parallel, each on its own smaller query, key and value projection. One head may follow the previous word, another a verb’s subject. The results are joined and passed through a linear layer.

head 1: previous word
head 2: content words
→ concatenate → linear
Two heads on the same five tokens learn different patterns: one looks at the word just before, the other at the content words. Hatched cells are blocked by the causal mask.

Masks decide who can see whom

Global attention lets every token read every other token. Causal attention hides later tokens: before the softmax, those scores are set to minus infinity, so their weight becomes zero and each row still sums to 1.

Pick a head and a mask in the lab. Rows are the token asking, columns the token being read.

A worked example

Take a model with d = 512 and 8 heads. Each head projects every token to queries, keys and values of 512 / 8 = 64 numbers, computes its own 64-number attention, and produces a 64-number output per token. The eight outputs are placed side by side, back to 512 numbers, and a final 512 × 512 linear layer mixes them. The cost is about the same as one big head, but the model gets eight different attention patterns.

Encoders, decoders and masks

The mask is just a table of 0s and 1s added before the softmax (as minus infinity where blocked). An encoder such as BERT uses no mask: every token sees the whole sentence. A decoder such as GPT uses the causal lower-triangular mask so that training on a full sentence never lets a position peek at the word it must predict.

Easy to confuse

Check yourself · 1 of 3

In causal attention, which cells of the attention matrix are blocked?

Check yourself · 2 of 3

Why use several heads instead of one big one?

Check yourself · 3 of 3

Which models use global (unmasked) attention?

Read more in the books

  • Build a Large Language Model (From Scratch), Sebastian Raschka: Ch. 3, “Coding Attention Mechanisms”: hiding future words with causal attention, and multi-head attention.
  • Dive into Deep Learning, Zhang, Lipton, Li, Smola: Ch. 11, “Attention Mechanisms and Transformers”: multi-head attention.
  • TinyML and Efficient Deep Learning Computing, Lecture 12: Transformer and LLM (slides), Song Han, MIT 6.5940: section I, multi-head attention and attention masking.
All books
Labmask and heads
How to read this
  • Pick a mask (Global or Causal) and a head. The big grid is that head’s attention matrix: row = token asking, column = token read, darker = more weight; numbers are the weights.
  • With the causal mask, cells above the diagonal are blocked (shown as dots) and each row still sums to 1.
  • Below: both heads side by side at small size, then how their outputs are joined: concatenated and mixed by a linear layer.
Mask
Head