Why transformers
Before transformers, text models read one token at a time or looked through a small window. Both limits shaped the design of attention.
Two older ways to read text
Recurrent networks (RNN, LSTM) read tokens in order and carry a small hidden state. They keep the order natural, but a link between two far tokens must pass through every step in between, and training cannot run those steps in parallel. Convolutional networks look at a window of neighbours: easy to parallelise, but each layer sees little context.
What attention changes
In self-attention, every token compares itself with every other token directly. The path between any two tokens has length one, and all comparisons happen in one matrix product. The price is that the number of pairs grows with the square of the length.
The lab counts steps for the three designs as the sequence grows.
A worked example
In “As a visual treat, the film is almost perfect”, deciding the sentiment of “film” needs “perfect” and “treat”, which sit several words away. An RNN must carry that information through every word in between; for a 100-token text, the first token reaches the last after 99 sequential steps. A CNN with a window of 3 widens its view by 2 tokens per layer, so it needs about 50 layers. Self-attention compares token 1 and token 100 directly, in one layer, and computes all such pairs at once on a GPU.
The cost
Comparing every pair is not free: for N tokens attention computes N² scores per head and layer, so doubling the context quadruples that work. Recurrent models cost only N steps but cannot be parallelised during training. The transformer trades extra arithmetic for parallelism and direct long-range links, a trade that GPUs reward.
Easy to confuse
- LSTMs fix part of the memory problem of plain RNNs with gates, but not the sequential dependency: step n still waits for step n − 1.
- Attention has no built-in sense of order; that is why transformers need positional encoding.
An RNN reads 100 tokens. How many sequential steps does information from token 1 need to reach token 100?
The hidden state is passed along one token at a time, so a link between positions i and j takes |i − j| steps, and the signal can fade on the way.
Why can a transformer train faster on a GPU than an RNN?
Attention compares all pairs of tokens in one matrix multiplication. An RNN’s state at step n depends on step n − 1, which forces step-by-step work.
What does a text CNN with kernel size 3 need in order to connect two tokens 10 positions apart?
Each layer sees only a few neighbours, so the reach grows slowly with depth. Attention reaches every token in one layer.
Read more in the books
- Deep Learning, Goodfellow, Bengio, Courville: Ch. 10, “Sequence Modeling: Recurrent and Recursive Nets”.
- Dive into Deep Learning, Zhang, Lipton, Li, Smola: Ch. 9, “Recurrent Neural Networks” and Ch. 10, “Modern Recurrent Neural Networks”.
- TinyML and Efficient Deep Learning Computing, Lecture 12: Transformer and LLM (slides), Song Han, MIT 6.5940: section I, pre-transformer era and the transformer.