FFN, GLU & normalisation
Between attention layers sits a small network, and around every sub-layer sits a normalisation. Small changes here improved every modern LLM.
The feed-forward network
After attention, each token’s vector goes through a two-layer network: widen from d to 4d, apply an activation, shrink back to d. It adds the non-linearity that attention lacks and holds most of the model’s parameters. ReLU cuts negatives to zero; GELU and Swish are smooth versions.
Gating and normalising
A gated linear unit multiplies one projection by a gate computed from another. SwiGLU uses Swish as the gate; to keep the parameter count equal, its hidden size is about 8/3 d. Around the layers, LayerNorm rescales each token, and a residual adds the input back. Pre-norm places the norm before the sub-layer; it is more stable than the original post-norm.
A worked example
At d = 4096, a plain FFN has two matrices of 4096 × 16384: about 134 million parameters. SwiGLU uses three matrices with hidden size 8/3 × 4096 ≈ 10923: 3 × 4096 × 10923 ≈ 134 million, the same budget. LLaMA rounds the hidden size to 11008. The difference is in the computation: one branch passes through Swish, the other stays linear, and their element-wise product lets the network switch features on and off per token.
Where the parameters live
In a standard block the FFN holds about two thirds of the parameters (8d² of roughly 12d²). That is why mixture-of-experts models replace exactly this layer with several copies: it is the biggest part, and each token only needs a little of it.
Easy to confuse
- The FFN works on each token separately; it never mixes information between tokens. That is attention’s job.
- GELU and Swish look alike: both are smooth and let small negative values through. SwiGLU is not an activation but a gated layer that uses Swish.
Why does a transformer block need a feed-forward network when attention is already there?
Attention reweights and sums value vectors, which is linear in the values. The FFN’s activation function adds the non-linearity needed to model more complex features, token by token.
What does LayerNorm normalise over?
LayerNorm computes the mean and spread across one token’s features, so it works the same for any batch size or sequence length. BatchNorm normalises across examples.
What is the difference between pre-norm and post-norm?
Pre-norm keeps a clean residual path from input to output, which makes deep stacks easier to train; most recent LLMs use it.
Read more in the books
- Build a Large Language Model (From Scratch), Sebastian Raschka: Ch. 4, “Implementing a GPT Model from Scratch”: layer normalization, GELU feed-forward network, shortcut connections.
- Deep Learning, Goodfellow, Bengio, Courville: Ch. 6, “Deep Feedforward Networks”: hidden units.
- TinyML and Efficient Deep Learning Computing, Lecture 12: Transformer and LLM (slides), Song Han, MIT 6.5940: section I, FFN and LayerNorm; section II, gated linear units.