Positional encoding
Attention has no sense of order. Positional encodings put it back, and the way they do it decides how well a model handles long text.
Absolute: add a position vector
The original transformer adds a vector for each position to the token embedding. With sines and cosines at different frequencies, every position gets a unique pattern. The position information then flows through the whole network inside Q, K and V.
Relative: work with distance
Relative schemes change the attention score instead. ALiBi subtracts a penalty that grows with distance. RoPE rotates each pair of query and key numbers by an angle that depends on position, so the score depends on the gap between tokens. Because they only see distance, relative schemes can sometimes run on sequences longer than those seen in training, and RoPE’s angles can be scaled down (position interpolation) to stretch the context window.
A worked example
With ALiBi and slope m = 0.5, a token attending to itself gets no penalty, to the token before it −0.5, to a token five places back −2.5. If two earlier tokens had the same raw score, the nearer one now wins. With RoPE, a query at position 9 and a key at position 7 give exactly the same score as a query at 2 and a key at 0, because only the gap of 2 matters.
Longer contexts
A model trained on 2,048 tokens has never seen larger rotation angles than those positions produce. Position interpolation divides the angles so that, for example, 8,192 positions are squeezed into the trained range; after a short fine-tune the model handles the longer context. This is how LLaMA-family models were stretched from 2K to 32K tokens and beyond.
Easy to confuse
- Absolute encodings change the input embeddings; ALiBi and RoPE change only the attention scores, and never the values.
- “Train short, test long” is a hope, not a guarantee: relative schemes help, but long-context quality usually still needs long-context training.
Why does a transformer need positional information at all?
Self-attention only compares tokens by content. Swap two tokens and every score is the same, just permuted, so the model cannot tell word order unless positions are injected.
What is special about RoPE’s dot product between a query at position m and a key at position n?
Rotating q by mθ and k by nθ means their angle differs by (m − n)θ. The score therefore depends on the relative distance, not on where the pair sits.
How does ALiBi add position information?
ALiBi leaves the embeddings alone and biases the attention scores by −slope × distance, so far tokens count less.
Read more in the books
- Dive into Deep Learning, Zhang, Lipton, Li, Smola: Ch. 11, “Attention Mechanisms and Transformers”: self-attention and positional encoding.
- Build a Large Language Model (From Scratch), Sebastian Raschka: Ch. 2, “Working with Text Data”: encoding word positions.
- Hands-On Large Language Models, Jay Alammar, Maarten Grootendorst: Ch. 3, looking inside large language models (positional embeddings and RoPE).
- TinyML and Efficient Deep Learning Computing, Lecture 12: Transformer and LLM (slides), Song Han, MIT 6.5940: section II, absolute and relative positional encoding.