LLM Lab

Derivatives & the chain rule

A derivative says how fast something changes. Training a network is repeatedly nudging weights in the direction the derivative says lowers the loss.

Where this maths shows up in a transformer
Embeddings (vectors)→Q·Kᵀ scores (matrices)→Softmax (probability)→Cross-entropy loss (logs)→Backprop (derivatives)→LayerNorm (mean, variance)

Slope at a point

The derivative f′(x) is the slope of the curve at x: how much f changes per small step in x. A positive slope means f is rising, a negative slope that it is falling, and zero means a flat spot such as a minimum. For a function of many inputs, the vector of slopes is the gradient.

The chain rule

If y = f(g(x)), then dy/dx = f′(g(x)) · g′(x): multiply the slopes along the chain. A neural network is one long chain, so backpropagation is the chain rule applied from the loss back to every weight.

Move x in the lab and compare the exact slope with a numerical check.

A worked example

At x = 1, u = 3·1 + 1 = 4 and y = 16. The local slopes are du/dx = 3 and dy/du = 2u = 8, so dy/dx = 8 × 3 = 24. Check it numerically: (f(1.001) − f(0.999)) / 0.002 = (16.024009 − 15.976009) / 0.002 = 24.0. Moving x a little to the right raises y about 24 times as much.

x1u = 3x + 1slope 3→u4y = u²slope 2u = 8→y16
backward: dy/dx = 8 × 3 = 24
The chain rule as a graph. Each arrow knows only its own local slope; multiplying them from y back to x gives the overall slope, 24. Backpropagation does this for every weight.

Many inputs: the gradient

A network’s loss depends on millions or billions of weights. The partial derivative for one weight is the slope when only that weight moves. Collect them all and you have the gradient, a vector with one entry per weight that points uphill. Gradient descent steps the other way: each weight moves by minus the learning rate times its entry.

Backpropagation

A network is a long chain of simple steps. Backpropagation runs the chain rule from the loss back to every weight, reusing the slopes already computed for the layers above, so the whole gradient costs only about twice a forward pass. The same rule explains a classic problem: multiplying many slopes smaller than 1 makes the product vanish, which is why deep recurrent networks struggled with long-range dependencies and why residual connections, which add a path with slope 1, help deep transformers train.

Easy to confuse

Check yourself · 1 of 3

The derivative of a loss with respect to a weight is positive. To lower the loss, the weight should…

Check yourself · 2 of 3

For y = (3x + 1)², what is dy/dx at x = 0?

Check yourself · 3 of 3

What does the chain rule let backpropagation do?

Read more in the books

  • Deep Learning, Goodfellow, Bengio, Courville: Ch. 4, “Numerical Computation”: gradient-based optimization.
  • Mathematics for Machine Learning, Deisenroth, Faisal, Ong: Ch. 5, “Vector Calculus”.
  • Dive into Deep Learning, Zhang, Lipton, Li, Smola: Ch. 2, “Preliminaries”: calculus.
All books
Labslope at a point
How to read this
  • The teal curve is y = (3x + 1)². The orange dot is the current x; the dashed orange line is the tangent there, and its slope is the derivative.
  • The row of circles is the computation graph x → u → y, with the current value inside each circle. Under each arrow is that step’s local slope.
  • The backward line multiplies the local slopes from y back to x. The product is dy/dx, which the box checks against a numerical estimate.