LLM Lab

Optimisers

Gradient descent says which way is down. Optimisers decide how big a step to take in each direction, and that decides how fast training goes.

Where this sits in the training loop
Data→Model makes a prediction→Loss measures the error→Gradients (backprop)→Update the weights (optimiser)

The narrow valley

Real loss surfaces are steep in some directions and nearly flat in others. Plain SGD uses one step size everywhere, so it bounces across the steep walls while crawling along the floor.

Momentum and Adam

Momentum keeps a velocity: a running average of past gradients. Adam also tracks the average squared gradient per parameter and divides by its square root, so every direction moves at a similar pace. Adam (and its variant AdamW) is the default for training transformers.

SGDw ← w − η·g
momentumv ← β·v + g; w ← w − η·v
Adamm ← β₁m + (1−β₁)g; s ← β₂s + (1−β₂)g²; w ← w − η·m / (√s + ε)
Three update rules (g is the gradient, η the learning rate). Momentum keeps a running direction; Adam also divides by a running size of the gradient, so every weight gets its own effective step. Typical values: β = 0.9, β₁ = 0.9, β₂ = 0.999.

Race the three from the same start on the lab’s valley x²/200 + y², minimum at the centre. Run 10 steps a few times: SGD stalls far from the minimum while the other two reach it.

A worked example

In a narrow valley the gradient points mostly across the valley, not along it. Plain SGD therefore zig-zags from wall to wall and creeps forward. Momentum averages the recent gradients: the across-valley parts cancel and the along-valley parts add up, so it speeds along. Adam divides each weight’s step by the running size of its gradient, so steep and shallow directions get similar step sizes.

In a language model

Almost every LLM is trained with Adam or AdamW (Adam with weight decay applied separately). Adam keeps two extra numbers per weight, which is why training needs much more memory than inference: for each fp16 weight there are typically fp32 copies of the weight and both moment estimates.

Easy to confuse

Check yourself · 1 of 3

Why does plain SGD zig-zag in a long, narrow valley?

Check yourself · 2 of 3

What does momentum add?

Check yourself · 3 of 3

Adam scales each parameter’s step by…

Read more in the books

  • Deep Learning, Goodfellow, Bengio, Courville: Ch. 8, “Optimization for Training Deep Models”, algorithms with adaptive learning rates.
  • Dive into Deep Learning, Zhang, Lipton, Li, Smola: Ch. 12, “Optimization Algorithms”, momentum, Adagrad, RMSProp and Adam.
  • Build a Large Language Model (From Scratch), Sebastian Raschka: Appendix D, “Adding Bells and Whistles to the Training Loop”.
All books
LabSGD vs Adam race
How to read this
  • The background is a long, narrow valley seen from above; the lowest point is in the middle. Each coloured path is one optimiser starting from the same point.
  • Run 10 steps advances all of them together; the learning-rate slider applies to all.
  • Watch plain SGD zig-zag across the valley while momentum and Adam move along it.