LLM families & Chinchilla
Large language models are transformers scaled up. The families differ in a few design choices, and in how much data they were fed.
Scale unlocks new behaviour
Model sizes grew from 0.1 billion parameters (2018) to hundreds of billions. At large sizes some abilities appear that smaller models lack (emergent abilities). GPT-3 showed that a 175-billion-parameter model can do new tasks zero-shot or few-shot from the prompt, with no fine-tuning: in-context learning.
How much data?
The Chinchilla law says that, for a fixed training budget, parameters and tokens should grow together, about 20 tokens per parameter. Open models such as LLaMA deliberately go far beyond that: a smaller model trained longer is cheaper to run.
A worked example
Chinchilla’s rule of thumb is about 20 training tokens per parameter. For a 7-billion-parameter model that is 140 billion tokens; LLaMA 7B was trained on 1 trillion, about 7 times more. GPT-3, with 175 billion parameters, saw 300 billion tokens, under 2 per parameter: by Chinchilla’s measure it was too large for its data. A smaller model trained longer reaches similar quality and costs much less to serve.
What the families share
Since GPT-3 almost all of these models are decoder-only transformers. The differences are in the details covered in this part: pre-norm and RMSNorm, SwiGLU feed-forward layers, RoPE positions, grouped-query attention, longer contexts and much more training data. Llama 3 trained on about 15 trillion tokens and was post-trained with supervised fine-tuning, rejection sampling and DPO.
Easy to confuse
- Chinchilla optimises the training budget only; when a model will serve many requests, training past the Chinchilla point is the cheaper choice overall.
- “Emergent” abilities depend partly on the metric: some look sudden only because the score is all-or-nothing.
What is few-shot (in-context) learning?
The examples only live in the context window. The weights stay the same; the large model infers the task from the pattern in the prompt.
Chinchilla suggests roughly how many training tokens per parameter for compute-optimal training?
The finding was that model size and data should scale together, at roughly 20 tokens per parameter. Earlier large models were trained on far fewer tokens than that.
Why train a small model on many more tokens than Chinchilla recommends?
Chinchilla optimises training cost only. If a model will serve millions of requests, spending more on training a smaller model reduces the total cost of ownership.
Read more in the books
- AI Engineering, Chip Huyen: Ch. 2, the sections on model size and scaling laws.
- Hands-On Large Language Models, Jay Alammar, Maarten Grootendorst: Ch. 1, an introduction to large language models.
- TinyML and Efficient Deep Learning Computing, Lecture 12: Transformer and LLM (slides), Song Han, MIT 6.5940: section III, large language models.