LLM Lab

Encoder, decoder, or both

The same attention blocks can be wired three ways. The wiring, and what the model is trained to guess, decide what it is good at.

Transformer design variants
Encoder / decoder→Positional encoding→KV cache: MHA → MQA → GQA→FFN → GLU

Three layouts

An encoder-only model (BERT) reads the whole text with global attention and outputs a vector per token: good for classification and search. A decoder-only model (GPT) uses causal attention and predicts the next token: it reads a prompt and keeps writing. An encoder-decoder model (T5, the original transformer) encodes the input, then a decoder writes the output while attending to the encoder.

BERT · encoder
GPT · decoder
T5 · encoder + decoder
Who may attend to whom (row = token asking, column = token read). Encoders see everything; decoders only see the past; T5 uses one of each, with cross-attention from decoder to encoder.

What they are trained to guess

BERT hides tokens and fills them in (masked language model), plus an optional next-sentence check, then is fine-tuned per task. GPT predicts the next word; small models are fine-tuned, while large ones can follow a task from a description (zero-shot) or a few examples (few-shot).

A worked example

For a review classifier, BERT reads “the film is almost perfect” with every token seeing every other, and a small classifier on top predicts “positive”. GPT reads the same text left to right and is best used to continue it, or to answer “Sentiment:” with the next token. T5 receives “sst2 sentence: the film is almost perfect” and writes “positive” as text. All three can do the task; their training decides which way is natural.

Why decoders won

Decoder-only models train on plain next-token prediction over any text, the same objective at every scale, and generate with the same weights they read with. GPT-3 showed that at 175 billion parameters such a model can follow a task from a few examples in the prompt, which removed the need for a separate fine-tuned model per task. Encoders remain common for search and classification, where a fixed vector per text is what you need.

Easy to confuse

Check yourself · 1 of 3

BERT is pretrained by hiding about 15% of the tokens and predicting them. What is that task called?

Check yourself · 2 of 3

Which layout do most modern chat LLMs use?

Check yourself · 3 of 3

T5 turns every task into…

Read more in the books

  • Dive into Deep Learning, Zhang, Lipton, Li, Smola: Ch. 11, “Attention Mechanisms and Transformers”: large-scale pretraining with transformers (encoder-only, encoder-decoder, decoder-only).
  • Hands-On Large Language Models, Jay Alammar, Maarten Grootendorst: Ch. 1, an introduction to large language models.
  • TinyML and Efficient Deep Learning Computing, Lecture 12: Transformer and LLM (slides), Song Han, MIT 6.5940: section II, encoder-decoder, encoder-only and decoder-only.
All books
Labpick the architecture
How to read this
  • Pick BERT, GPT or T5. The 5 × 5 grids show who may attend to whom (teal = allowed): one grid for an encoder or decoder, two for T5.
  • The chips below show the pretraining objective with a real sentence: BERT fills a [MASK], GPT predicts the next word, T5 maps an input text to an output text.
  • The task buttons pick a task; the answer names the layout that suits it and why.
Which model fits the task?