Encoder, decoder, or both
The same attention blocks can be wired three ways. The wiring, and what the model is trained to guess, decide what it is good at.
Three layouts
An encoder-only model (BERT) reads the whole text with global attention and outputs a vector per token: good for classification and search. A decoder-only model (GPT) uses causal attention and predicts the next token: it reads a prompt and keeps writing. An encoder-decoder model (T5, the original transformer) encodes the input, then a decoder writes the output while attending to the encoder.
What they are trained to guess
BERT hides tokens and fills them in (masked language model), plus an optional next-sentence check, then is fine-tuned per task. GPT predicts the next word; small models are fine-tuned, while large ones can follow a task from a description (zero-shot) or a few examples (few-shot).
A worked example
For a review classifier, BERT reads “the film is almost perfect” with every token seeing every other, and a small classifier on top predicts “positive”. GPT reads the same text left to right and is best used to continue it, or to answer “Sentiment:” with the next token. T5 receives “sst2 sentence: the film is almost perfect” and writes “positive” as text. All three can do the task; their training decides which way is natural.
Why decoders won
Decoder-only models train on plain next-token prediction over any text, the same objective at every scale, and generate with the same weights they read with. GPT-3 showed that at 175 billion parameters such a model can follow a task from a few examples in the prompt, which removed the need for a separate fine-tuned model per task. Encoders remain common for search and classification, where a fixed vector per text is what you need.
Easy to confuse
- BERT is not a chat model: it outputs vectors, not text, and needs a task-specific head.
- “Encoder” and “decoder” describe the mask and the objective, not different kinds of blocks; the blocks themselves are almost identical.
BERT is pretrained by hiding about 15% of the tokens and predicting them. What is that task called?
Masked language modelling forces the model to use context on both sides of a gap, which suits an encoder with global attention.
Which layout do most modern chat LLMs use?
A decoder-only transformer with causal attention reads the prompt and continues it with the same weights. It is simple to scale and handles many tasks as text continuation.
T5 turns every task into…
T5 writes the task as text (“translate English to German: …”) and answers as text, so one model and one loss cover translation, summarisation and more.
Read more in the books
- Dive into Deep Learning, Zhang, Lipton, Li, Smola: Ch. 11, “Attention Mechanisms and Transformers”: large-scale pretraining with transformers (encoder-only, encoder-decoder, decoder-only).
- Hands-On Large Language Models, Jay Alammar, Maarten Grootendorst: Ch. 1, an introduction to large language models.
- TinyML and Efficient Deep Learning Computing, Lecture 12: Transformer and LLM (slides), Song Han, MIT 6.5940: section II, encoder-decoder, encoder-only and decoder-only.