LLM Lab

Positional encoding

Attention has no sense of order. Positional encodings put it back, and the way they do it decides how well a model handles long text.

Where this sits in a transformer
Tokenizer→Embedding→Positional encoding→Multi-head attention→Feed-forward→LayerNorm + residual→Linear head→Sampling
Transformer design variants
Encoder / decoder→Positional encoding→KV cache: MHA → MQA → GQA→FFN → GLU

Absolute: add a position vector

The original transformer adds a vector for each position to the token embedding. With sines and cosines at different frequencies, every position gets a unique pattern. The position information then flows through the whole network inside Q, K and V.

Relative: work with distance

Relative schemes change the attention score instead. ALiBi subtracts a penalty that grows with distance. RoPE rotates each pair of query and key numbers by an angle that depends on position, so the score depends on the gap between tokens. Because they only see distance, relative schemes can sometimes run on sequences longer than those seen in training, and RoPE’s angles can be scaled down (position interpolation) to stretch the context window.

q@2k@0q@5k@3q@9k@7
RoPE rotates a pair of query numbers by m·θ and a pair of key numbers by n·θ. In all three dials the gap m − n is 2, so the angle between query and key, and with it the attention score, is the same.

A worked example

With ALiBi and slope m = 0.5, a token attending to itself gets no penalty, to the token before it −0.5, to a token five places back −2.5. If two earlier tokens had the same raw score, the nearer one now wins. With RoPE, a query at position 9 and a key at position 7 give exactly the same score as a query at 2 and a key at 0, because only the gap of 2 matters.

Longer contexts

A model trained on 2,048 tokens has never seen larger rotation angles than those positions produce. Position interpolation divides the angles so that, for example, 8,192 positions are squeezed into the trained range; after a short fine-tune the model handles the longer context. This is how LLaMA-family models were stretched from 2K to 32K tokens and beyond.

Easy to confuse

Check yourself · 1 of 3

Why does a transformer need positional information at all?

Check yourself · 2 of 3

What is special about RoPE’s dot product between a query at position m and a key at position n?

Check yourself · 3 of 3

How does ALiBi add position information?

Read more in the books

  • Dive into Deep Learning, Zhang, Lipton, Li, Smola: Ch. 11, “Attention Mechanisms and Transformers”: self-attention and positional encoding.
  • Build a Large Language Model (From Scratch), Sebastian Raschka: Ch. 2, “Working with Text Data”: encoding word positions.
  • Hands-On Large Language Models, Jay Alammar, Maarten Grootendorst: Ch. 3, looking inside large language models (positional embeddings and RoPE).
  • TinyML and Efficient Deep Learning Computing, Lecture 12: Transformer and LLM (slides), Song Han, MIT 6.5940: section II, absolute and relative positional encoding.
All books
Labrotate by position
How to read this
  • Pick a scheme. RoPE: two arrows, a query at position m and a key at position n, each rotated by its position. Move m and n, then “shift both”: the score stays the same while m − n is unchanged.
  • ALiBi: three grids, the raw scores, the bias −m·(i − j), and their sum. Orange means a penalty; hatched cells are masked (later tokens).
  • Sinusoidal: one row per position t, one column per sine or cosine feature; teal positive, orange negative.
Scheme