Optimisers
Gradient descent says which way is down. Optimisers decide how big a step to take in each direction, and that decides how fast training goes.
The narrow valley
Real loss surfaces are steep in some directions and nearly flat in others. Plain SGD uses one step size everywhere, so it bounces across the steep walls while crawling along the floor.
Momentum and Adam
Momentum keeps a velocity: a running average of past gradients. Adam also tracks the average squared gradient per parameter and divides by its square root, so every direction moves at a similar pace. Adam (and its variant AdamW) is the default for training transformers.
Race the three from the same start on the lab’s valley x²/200 + y², minimum at the centre. Run 10 steps a few times: SGD stalls far from the minimum while the other two reach it.
A worked example
In a narrow valley the gradient points mostly across the valley, not along it. Plain SGD therefore zig-zags from wall to wall and creeps forward. Momentum averages the recent gradients: the across-valley parts cancel and the along-valley parts add up, so it speeds along. Adam divides each weight’s step by the running size of its gradient, so steep and shallow directions get similar step sizes.
In a language model
Almost every LLM is trained with Adam or AdamW (Adam with weight decay applied separately). Adam keeps two extra numbers per weight, which is why training needs much more memory than inference: for each fp16 weight there are typically fp32 copies of the weight and both moment estimates.
Easy to confuse
- The optimiser does not compute gradients; backpropagation does. The optimiser turns gradients into updates.
- A good learning rate depends on the optimiser: values that work for Adam are usually far smaller than those for plain SGD.
Why does plain SGD zig-zag in a long, narrow valley?
In the steep direction it overshoots back and forth; in the shallow direction it barely moves.
What does momentum add?
The velocity accumulates along the valley floor while the side-to-side components average out.
Adam scales each parameter’s step by…
Dividing by √(average squared gradient) gives large steps in flat directions and small ones in steep directions.
Read more in the books
- Deep Learning, Goodfellow, Bengio, Courville: Ch. 8, “Optimization for Training Deep Models”, algorithms with adaptive learning rates.
- Dive into Deep Learning, Zhang, Lipton, Li, Smola: Ch. 12, “Optimization Algorithms”, momentum, Adagrad, RMSProp and Adam.
- Build a Large Language Model (From Scratch), Sebastian Raschka: Appendix D, “Adding Bells and Whistles to the Training Loop”.