LLM Lab

Transformer block

A transformer is one small block repeated many times. Learn the block and you know the model.

Where this sits in a transformer
Tokenizer→Embedding→Positional encoding→Multi-head attention→Feed-forward→LayerNorm + residual→Linear head→Sampling

Attention mixes, the network thinks

Each token starts as a vector: its embedding plus a vector for its position. A block then does two things. Self-attention lets every token gather information from the tokens before it. A feed-forward network then transforms each token’s vector on its own: it widens the vector from d to 4d, applies an activation, and narrows it back to d (an “inverted bottleneck”).

Embed + position→LayerNorm→Multi-head attention→+ residual→LayerNorm→FFN d → 4d → d→+ residual
in
FFN
out
One pre-norm block. Every token’s vector keeps the same width d on the way in and out; only inside the feed-forward network does it widen to 4d (16 cells here).

Add and normalise

Around both steps sits a residual connection (add the input back) and a layer norm (rescale each token’s vector). The original design normalises after each add (post-norm); most models now normalise before each sub-layer (pre-norm), which trains more stably.

Step through the block in the lab for “the cat sat”. Every row is one token, in the same colour all the way through, and every vector keeps the same width d = 4 until the feed-forward layer briefly widens it.

Counting the parameters

A block with width d has about 4d² parameters in attention (the Q, K, V and output projections) and 8d² in the feed-forward network (d × 4d and 4d × d), about 12d² in all. GPT-2 small has 12 blocks with d = 768: 12 × 12 × 768² ≈ 85 million, plus about 39 million in the token and position embeddings, for its 124 million total. A 7-billion-parameter LLaMA stacks 32 blocks with d = 4096.

Easy to confuse

Check yourself · 1 of 3

What does the residual (“add”) connection do in a block?

Check yourself · 2 of 3

Which part of the block lets one token look at other tokens?

Check yourself · 3 of 3

A model has 24 identical blocks stacked. What flows from one block to the next?

Read more in the books

  • Build a Large Language Model (From Scratch), Sebastian Raschka: Ch. 4, “Implementing a GPT Model from Scratch”.
  • Dive into Deep Learning, Zhang, Lipton, Li, Smola: Ch. 11, “Attention Mechanisms and Transformers”, the transformer architecture.
  • Hands-On Large Language Models, Jay Alammar, Maarten Grootendorst: Ch. 3, looking inside large language models.
All books
Labstep through
How to read this
  • Each row is a token, in the same colour as everywhere on the site. Each strip is its vector: d = 4 numbers, teal positive, orange negative.
  • The stage map shows the six stages of the block and lights the current one; the card below explains it. At step 4 the strip widens to 16 cells, the 4d hidden layer of the feed-forward network.
  • Switch Pre-norm and Post-norm to see where LayerNorm sits. The numbers change because the order of the operations changes.
Norm placement