Gradient descent
Training a model means finding the weights with the lowest loss. Gradient descent gets there by walking downhill, one small step at a time.
Follow the slope
The loss measures how wrong the model is for a given weight w. Its gradient (the slope) says which way the loss rises. Step the other way:
w ← w − η · ∂loss/∂w
The learning rate η sets the step size. The lab uses a single weight and the loss (w − 3)² + 1, whose minimum is at w = 3.
Too small, too large
A tiny learning rate crawls. A large one overshoots the minimum and bounces from side to side; past a threshold each bounce is bigger than the last and training diverges. Real models have millions of weights, but each one moves by this same rule.
A worked example
The lab’s loss is (w − 3)² + 1, with slope 2(w − 3). From w = −1.5 the slope is −9, so with a learning rate of 0.1 the next w is −1.5 − 0.1 × (−9) = −0.6. Each step here cuts the distance to the minimum at w = 3 by 20%. With a learning rate of 1.0 the step overshoots to the mirror point and bounces forever; above 1.0 it diverges.
In a language model
A transformer has billions of weights, so the gradient is a vector with billions of entries, computed by backpropagation on a batch of text. Training takes hundreds of thousands of such steps. The learning rate is usually warmed up from zero and then decayed, because large steps early in training or late in training are what cause loss spikes and divergence.
Easy to confuse
- The gradient points uphill; descent moves the other way, hence the minus sign.
- Stochastic gradient descent uses a small batch of data for each step, so each gradient is a noisy estimate; that noise is expected, not a bug.
The gradient at the current point is positive. Which way does gradient descent move w?
The update is w ← w − η·gradient. A positive slope means the loss rises to the right, so the step goes left.
For this loss, what happens with a learning rate of 1.05?
Each step multiplies the distance to the minimum by (1 − 2η). Once η > 1 that factor is below −1, so every jump lands further away on the other side.
Why does the step get smaller as w nears the minimum, even with a fixed learning rate?
The step is η times the slope, and the slope flattens to zero at the bottom of the bowl.
Read more in the books
- Deep Learning, Goodfellow, Bengio, Courville: Ch. 4, “Numerical Computation”, section 4.3 (gradient-based optimization).
- Dive into Deep Learning, Zhang, Lipton, Li, Smola: Ch. 12, “Optimization Algorithms”, gradient descent.
- Hands-On Machine Learning with Scikit-Learn, Keras and TensorFlow (3rd ed.), Aurélien Géron: Ch. 4, “Training Models”.