Transformer block
A transformer is one small block repeated many times. Learn the block and you know the model.
Attention mixes, the network thinks
Each token starts as a vector: its embedding plus a vector for its position. A block then does two things. Self-attention lets every token gather information from the tokens before it. A feed-forward network then transforms each token’s vector on its own: it widens the vector from d to 4d, applies an activation, and narrows it back to d (an “inverted bottleneck”).
Add and normalise
Around both steps sits a residual connection (add the input back) and a layer norm (rescale each token’s vector). The original design normalises after each add (post-norm); most models now normalise before each sub-layer (pre-norm), which trains more stably.
Step through the block in the lab for “the cat sat”. Every row is one token, in the same colour all the way through, and every vector keeps the same width d = 4 until the feed-forward layer briefly widens it.
Counting the parameters
A block with width d has about 4d² parameters in attention (the Q, K, V and output projections) and 8d² in the feed-forward network (d × 4d and 4d × d), about 12d² in all. GPT-2 small has 12 blocks with d = 768: 12 × 12 × 768² ≈ 85 million, plus about 39 million in the token and position embeddings, for its 124 million total. A 7-billion-parameter LLaMA stacks 32 blocks with d = 4096.
Easy to confuse
- The blocks share their design but not their weights: each of the 32 blocks has its own matrices.
- The vector that flows through the residual connections, the “residual stream”, is what each block reads from and writes back to; the layers add changes to it rather than replacing it.
What does the residual (“add”) connection do in a block?
Each sub-layer returns x + f(x). The input flows straight through, which keeps gradients healthy in deep stacks and lets a layer start as “do nothing”.
Which part of the block lets one token look at other tokens?
Attention is the only step that mixes information across positions. The feed-forward network and the norm act on each token on its own.
A model has 24 identical blocks stacked. What flows from one block to the next?
A block maps a sequence of vectors to a sequence of vectors of the same shape, which is why blocks can be stacked as many times as you like.
Read more in the books
- Build a Large Language Model (From Scratch), Sebastian Raschka: Ch. 4, “Implementing a GPT Model from Scratch”.
- Dive into Deep Learning, Zhang, Lipton, Li, Smola: Ch. 11, “Attention Mechanisms and Transformers”, the transformer architecture.
- Hands-On Large Language Models, Jay Alammar, Maarten Grootendorst: Ch. 3, looking inside large language models.