LLM Lab

Why transformers

Before transformers, text models read one token at a time or looked through a small window. Both limits shaped the design of attention.

Where this sits in a transformer
Tokenizer→Embedding→Positional encoding→Multi-head attention→Feed-forward→LayerNorm + residual→Linear head→Sampling

Two older ways to read text

Recurrent networks (RNN, LSTM) read tokens in order and carry a small hidden state. They keep the order natural, but a link between two far tokens must pass through every step in between, and training cannot run those steps in parallel. Convolutional networks look at a window of neighbours: easy to parallelise, but each layer sees little context.

RNN / LSTM99 sequential steps between token 1 and token 100
CNN, kernel 350 stacked layers to connect them
Self-attention1 layer: every pair is compared directly
How far apart are token 1 and token 100? An RNN passes information one step at a time, a CNN widens its window layer by layer, and attention links them in one step at the price of comparing all N² pairs.

What attention changes

In self-attention, every token compares itself with every other token directly. The path between any two tokens has length one, and all comparisons happen in one matrix product. The price is that the number of pairs grows with the square of the length.

The lab counts steps for the three designs as the sequence grows.

A worked example

In “As a visual treat, the film is almost perfect”, deciding the sentiment of “film” needs “perfect” and “treat”, which sit several words away. An RNN must carry that information through every word in between; for a 100-token text, the first token reaches the last after 99 sequential steps. A CNN with a window of 3 widens its view by 2 tokens per layer, so it needs about 50 layers. Self-attention compares token 1 and token 100 directly, in one layer, and computes all such pairs at once on a GPU.

The cost

Comparing every pair is not free: for N tokens attention computes N² scores per head and layer, so doubling the context quadruples that work. Recurrent models cost only N steps but cannot be parallelised during training. The transformer trades extra arithmetic for parallelism and direct long-range links, a trade that GPUs reward.

Easy to confuse

Check yourself · 1 of 3

An RNN reads 100 tokens. How many sequential steps does information from token 1 need to reach token 100?

Check yourself · 2 of 3

Why can a transformer train faster on a GPU than an RNN?

Check yourself · 3 of 3

What does a text CNN with kernel size 3 need in order to connect two tokens 10 positions apart?

Read more in the books

  • Deep Learning, Goodfellow, Bengio, Courville: Ch. 10, “Sequence Modeling: Recurrent and Recursive Nets”.
  • Dive into Deep Learning, Zhang, Lipton, Li, Smola: Ch. 9, “Recurrent Neural Networks” and Ch. 10, “Modern Recurrent Neural Networks”.
  • TinyML and Efficient Deep Learning Computing, Lecture 12: Transformer and LLM (slides), Song Han, MIT 6.5940: section I, pre-transformer era and the transformer.
All books
Labsteps to connect tokens
How to read this
  • Top: the sentence as coloured tokens. Pick RNN, CNN or Attention to see how information reaches the last token “is”. Numbers on the RNN arrows count the sequential steps.
  • Below: drag the sequence length and kernel size. Each bar is the number of steps (or layers) needed to connect the first and last token; the line under it says whether those steps must run in order.