LLM Lab

FFN, GLU & normalisation

Between attention layers sits a small network, and around every sub-layer sits a normalisation. Small changes here improved every modern LLM.

Where this sits in a transformer
Tokenizer→Embedding→Positional encoding→Multi-head attention→Feed-forward→LayerNorm + residual→Linear head→Sampling
Transformer design variants
Encoder / decoder→Positional encoding→KV cache: MHA → MQA → GQA→FFN → GLU

The feed-forward network

After attention, each token’s vector goes through a two-layer network: widen from d to 4d, apply an activation, shrink back to d. It adds the non-linearity that attention lacks and holds most of the model’s parameters. ReLU cuts negatives to zero; GELU and Swish are smooth versions.

Gating and normalising

A gated linear unit multiplies one projection by a gate computed from another. SwiGLU uses Swish as the gate; to keep the parameter count equal, its hidden size is about 8/3 d. Around the layers, LayerNorm rescales each token, and a residual adds the input back. Pre-norm places the norm before the sub-layer; it is more stable than the original post-norm.

FFN → 4d → d
SwiGLU → 8/3 d · Swish ⊙ 8/3 d → d
The plain FFN widens to 4d and back (two matrices). SwiGLU computes two 8/3 d branches, multiplies them element by element, and projects back (three matrices, about the same number of parameters).

A worked example

At d = 4096, a plain FFN has two matrices of 4096 × 16384: about 134 million parameters. SwiGLU uses three matrices with hidden size 8/3 × 4096 ≈ 10923: 3 × 4096 × 10923 ≈ 134 million, the same budget. LLaMA rounds the hidden size to 11008. The difference is in the computation: one branch passes through Swish, the other stays linear, and their element-wise product lets the network switch features on and off per token.

Where the parameters live

In a standard block the FFN holds about two thirds of the parameters (8d² of roughly 12d²). That is why mixture-of-experts models replace exactly this layer with several copies: it is the biggest part, and each token only needs a little of it.

Easy to confuse

Check yourself · 1 of 3

Why does a transformer block need a feed-forward network when attention is already there?

Check yourself · 2 of 3

What does LayerNorm normalise over?

Check yourself · 3 of 3

What is the difference between pre-norm and post-norm?

Read more in the books

  • Build a Large Language Model (From Scratch), Sebastian Raschka: Ch. 4, “Implementing a GPT Model from Scratch”: layer normalization, GELU feed-forward network, shortcut connections.
  • Deep Learning, Goodfellow, Bengio, Courville: Ch. 6, “Deep Feedforward Networks”: hidden units.
  • TinyML and Efficient Deep Learning Computing, Lecture 12: Transformer and LLM (slides), Song Han, MIT 6.5940: section I, FFN and LayerNorm; section II, gated linear units.
All books
Labactivation plots
How to read this
  • The plot shows the chosen activation function f(x) for x from −4 to 4.
  • The tiles count parameters of one feed-forward layer at width d: the plain FFN against SwiGLU with hidden size 8/3 d. The bars below draw both shapes.
  • Pre-norm / Post-norm writes out the order of operations inside one block.
Activation
FFN (d → 4d → d)–
SwiGLU (hidden 8/3 d)–
Norm placement