LLM Lab

LLM families & Chinchilla

Large language models are transformers scaled up. The families differ in a few design choices, and in how much data they were fed.

Where this sits in the life of an LLM
Pretraining at scale→Supervised fine-tuning→Preference tuning (RLHF, DPO)→Serving (KV cache, quantisation)

Scale unlocks new behaviour

Model sizes grew from 0.1 billion parameters (2018) to hundreds of billions. At large sizes some abilities appear that smaller models lack (emergent abilities). GPT-3 showed that a 175-billion-parameter model can do new tasks zero-shot or few-shot from the prompt, with no fine-tuning: in-context learning.

How much data?

The Chinchilla law says that, for a fixed training budget, parameters and tokens should grow together, about 20 tokens per parameter. Open models such as LLaMA deliberately go far beyond that: a smaller model trained longer is cheaper to run.

A worked example

Chinchilla’s rule of thumb is about 20 training tokens per parameter. For a 7-billion-parameter model that is 140 billion tokens; LLaMA 7B was trained on 1 trillion, about 7 times more. GPT-3, with 175 billion parameters, saw 300 billion tokens, under 2 per parameter: by Chinchilla’s measure it was too large for its data. A smaller model trained longer reaches similar quality and costs much less to serve.

GPT-3 · 175Btrained on 300B Chinchilla: 3.5T
LLaMA · 7Btrained on 1T Chinchilla: 140B
Llama 2 · 70Btrained on 2T Chinchilla: 1.4T
Training tokens (teal) against Chinchilla’s ~20 tokens per parameter (grey), on a log scale. GPT-3 was under-trained by that measure; LLaMA models deliberately go well past it to be cheaper to run.

What the families share

Since GPT-3 almost all of these models are decoder-only transformers. The differences are in the details covered in this part: pre-norm and RMSNorm, SwiGLU feed-forward layers, RoPE positions, grouped-query attention, longer contexts and much more training data. Llama 3 trained on about 15 trillion tokens and was post-trained with supervised fine-tuning, rejection sampling and DPO.

Easy to confuse

Check yourself · 1 of 3

What is few-shot (in-context) learning?

Check yourself · 2 of 3

Chinchilla suggests roughly how many training tokens per parameter for compute-optimal training?

Check yourself · 3 of 3

Why train a small model on many more tokens than Chinchilla recommends?

Read more in the books

  • AI Engineering, Chip Huyen: Ch. 2, the sections on model size and scaling laws.
  • Hands-On Large Language Models, Jay Alammar, Maarten Grootendorst: Ch. 1, an introduction to large language models.
  • TinyML and Efficient Deep Learning Computing, Lecture 12: Transformer and LLM (slides), Song Han, MIT 6.5940: section III, large language models.
All books
Labcompare the models
How to read this
  • The chart plots model size by year on a log scale: each step up is ten times larger.
  • Pick a model to see its configuration. The two tiles compare its training tokens per parameter with Chinchilla’s 20, and the note says whether it was trained past that point.
Tokens per parameter–
Chinchilla (20 per parameter)–