Pretraining & scaling laws
Pretraining loss falls smoothly and predictably as models, data and compute grow. That predictability is what lets labs plan billion-dollar training runs.
Loss as a formula
Hoffmann et al. (2022) fit the final loss of many training runs with L(N, D) = E + A/N^α + B/D^β, where N is the number of parameters and D the number of training tokens. E is the loss no model can beat; the other two terms shrink as the model or the data grows.
Spending a budget
Training compute is roughly C ≈ 6·N·D FLOPs. For a fixed C, a bigger model sees fewer tokens. The lab plots the fitted loss across model sizes at your budget (an “isoFLOP” curve) and marks the best split. The constants come from a 2024 replication (Besiroglu et al.) that refitted the original paper’s data; treat the outputs as illustrative.
A worked example
The Chinchilla fit predicts loss as L = E + A / N^α + B / D^β, with N parameters and D training tokens; the original paper’s constants (the lab uses the 2024 refit) are about E = 1.69, A = 406.4, B = 410.7, α = 0.34 and β = 0.28. For N = 70 billion and D = 1.4 trillion: A / N^α ≈ 0.08 and B / D^β ≈ 0.16, so L ≈ 1.94. Training compute is about 6 × N × D ≈ 5.9 × 10²³ FLOPs.
In practice
Labs fit such curves on small models and use them to choose the size of the big run before spending the budget. Because the curves are smooth, they also predict what a bigger model will reach. Later models, such as LLaMA, chose smaller sizes trained on far more tokens than this optimum, because a smaller model is cheaper every time it is used.
Easy to confuse
- A lower loss does not translate evenly into every skill; some abilities improve smoothly, others appear only past a certain scale.
- The law describes pretraining loss, not chat quality after fine-tuning and preference tuning.
With a fixed compute budget, a much bigger model trained on fewer tokens…
Compute ≈ 6·N·D. Spending it all on parameters starves the model of data; the loss curve over model size has a minimum.
Roughly how many training tokens per parameter did the Chinchilla analysis find compute-optimal?
Hoffmann et al. (2022) found parameters and tokens should grow together, around 20 tokens per parameter.
Why train “past” the compute-optimal point in practice?
Compute-optimal minimises training cost for a given loss; inference cost favours smaller models trained on more data.
Read more in the books
- AI Engineering, Chip Huyen: Ch. 2, the section on scaling laws and compute-optimal models.
- Build a Large Language Model (From Scratch), Sebastian Raschka: Ch. 5, “Pretraining on Unlabeled Data”.