LLM Lab

Pretraining & scaling laws

Pretraining loss falls smoothly and predictably as models, data and compute grow. That predictability is what lets labs plan billion-dollar training runs.

Where this sits in the life of an LLM
Pretraining at scale→Supervised fine-tuning→Preference tuning (RLHF, DPO)→Serving (KV cache, quantisation)

Loss as a formula

Hoffmann et al. (2022) fit the final loss of many training runs with L(N, D) = E + A/N^α + B/D^β, where N is the number of parameters and D the number of training tokens. E is the loss no model can beat; the other two terms shrink as the model or the data grows.

Spending a budget

Training compute is roughly C ≈ 6·N·D FLOPs. For a fixed C, a bigger model sees fewer tokens. The lab plots the fitted loss across model sizes at your budget (an “isoFLOP” curve) and marks the best split. The constants come from a 2024 replication (Besiroglu et al.) that refitted the original paper’s data; treat the outputs as illustrative.

A worked example

The Chinchilla fit predicts loss as L = E + A / N^α + B / D^β, with N parameters and D training tokens; the original paper’s constants (the lab uses the 2024 refit) are about E = 1.69, A = 406.4, B = 410.7, α = 0.34 and β = 0.28. For N = 70 billion and D = 1.4 trillion: A / N^α ≈ 0.08 and B / D^β ≈ 0.16, so L ≈ 1.94. Training compute is about 6 × N × D ≈ 5.9 × 10²³ FLOPs.

irreducible E1.69
model size term A / N^α0.084
data term B / D^β0.163
predicted loss≈ 1.94
Chinchilla’s fitted law for a 70-billion-parameter model trained on 1.4 trillion tokens. Most of the loss is the irreducible part; more parameters shrink the second term, more data the third.

In practice

Labs fit such curves on small models and use them to choose the size of the big run before spending the budget. Because the curves are smooth, they also predict what a bigger model will reach. Later models, such as LLaMA, chose smaller sizes trained on far more tokens than this optimum, because a smaller model is cheaper every time it is used.

Easy to confuse

Check yourself · 1 of 3

With a fixed compute budget, a much bigger model trained on fewer tokens…

Check yourself · 2 of 3

Roughly how many training tokens per parameter did the Chinchilla analysis find compute-optimal?

Check yourself · 3 of 3

Why train “past” the compute-optimal point in practice?

Read more in the books

  • AI Engineering, Chip Huyen: Ch. 2, the section on scaling laws and compute-optimal models.
  • Build a Large Language Model (From Scratch), Sebastian Raschka: Ch. 5, “Pretraining on Unlabeled Data”.
All books
Labscaling curves
How to read this
  • The slider sets a fixed compute budget (in FLOPs, log scale).
  • The curve shows the predicted loss for different model sizes trained with that budget: a small model sees lots of data, a large one little. The lowest point is the compute-optimal size.
  • The readout reports that size and the number of training tokens it implies.