LLM Lab

Gradient descent

Training a model means finding the weights with the lowest loss. Gradient descent gets there by walking downhill, one small step at a time.

Where this sits in the training loop
Data→Model makes a prediction→Loss measures the error→Gradients (backprop)→Update the weights (optimiser)

Follow the slope

The loss measures how wrong the model is for a given weight w. Its gradient (the slope) says which way the loss rises. Step the other way:

w ← w − η · ∂loss/∂w

The learning rate η sets the step size. The lab uses a single weight and the loss (w − 3)² + 1, whose minimum is at w = 3.

Too small, too large

A tiny learning rate crawls. A large one overshoots the minimum and bounces from side to side; past a threshold each bounce is bigger than the last and training diverges. Real models have millions of weights, but each one moves by this same rule.

A worked example

The lab’s loss is (w − 3)² + 1, with slope 2(w − 3). From w = −1.5 the slope is −9, so with a learning rate of 0.1 the next w is −1.5 − 0.1 × (−9) = −0.6. Each step here cuts the distance to the minimum at w = 3 by 20%. With a learning rate of 1.0 the step overshoots to the mirror point and bounces forever; above 1.0 it diverges.

step 1w ← -1.50 − 0.1 × -9.00 = -0.603.60 from the minimum
step 2w ← -0.60 − 0.1 × -7.20 = 0.122.88 from the minimum
step 3w ← 0.12 − 0.1 × -5.76 = 0.702.30 from the minimum
step 4w ← 0.70 − 0.1 × -4.61 = 1.161.84 from the minimum
step 5w ← 1.16 − 0.1 × -3.69 = 1.531.47 from the minimum
Gradient descent on loss (w − 3)² + 1, starting at w = −1.5 with learning rate 0.1. Each step moves against the slope; the distance to the minimum at w = 3 shrinks by 20% per step.

In a language model

A transformer has billions of weights, so the gradient is a vector with billions of entries, computed by backpropagation on a batch of text. Training takes hundreds of thousands of such steps. The learning rate is usually warmed up from zero and then decayed, because large steps early in training or late in training are what cause loss spikes and divergence.

Easy to confuse

Check yourself · 1 of 3

The gradient at the current point is positive. Which way does gradient descent move w?

Check yourself · 2 of 3

For this loss, what happens with a learning rate of 1.05?

Check yourself · 3 of 3

Why does the step get smaller as w nears the minimum, even with a fixed learning rate?

Read more in the books

  • Deep Learning, Goodfellow, Bengio, Courville: Ch. 4, “Numerical Computation”, section 4.3 (gradient-based optimization).
  • Dive into Deep Learning, Zhang, Lipton, Li, Smola: Ch. 12, “Optimization Algorithms”, gradient descent.
  • Hands-On Machine Learning with Scikit-Learn, Keras and TensorFlow (3rd ed.), Aurélien Géron: Ch. 4, “Training Models”.
All books
Lablearning-rate slider
How to read this
  • The curve is the loss as a function of one weight w; the ball is the current w and the faded dots are its recent positions.
  • Step moves once, Run 10 steps moves ten times, Reset starts again. The learning-rate slider sets how big each step is.
  • The stats line shows w, the loss and the gradient; the box below writes out the last update w ← w − learning rate × gradient.
w = 3