Derivatives & the chain rule
A derivative says how fast something changes. Training a network is repeatedly nudging weights in the direction the derivative says lowers the loss.
Slope at a point
The derivative f′(x) is the slope of the curve at x: how much f changes per small step in x. A positive slope means f is rising, a negative slope that it is falling, and zero means a flat spot such as a minimum. For a function of many inputs, the vector of slopes is the gradient.
The chain rule
If y = f(g(x)), then dy/dx = f′(g(x)) · g′(x): multiply the slopes along the chain. A neural network is one long chain, so backpropagation is the chain rule applied from the loss back to every weight.
Move x in the lab and compare the exact slope with a numerical check.
A worked example
At x = 1, u = 3·1 + 1 = 4 and y = 16. The local slopes are du/dx = 3 and dy/du = 2u = 8, so dy/dx = 8 × 3 = 24. Check it numerically: (f(1.001) − f(0.999)) / 0.002 = (16.024009 − 15.976009) / 0.002 = 24.0. Moving x a little to the right raises y about 24 times as much.
Many inputs: the gradient
A network’s loss depends on millions or billions of weights. The partial derivative for one weight is the slope when only that weight moves. Collect them all and you have the gradient, a vector with one entry per weight that points uphill. Gradient descent steps the other way: each weight moves by minus the learning rate times its entry.
Backpropagation
A network is a long chain of simple steps. Backpropagation runs the chain rule from the loss back to every weight, reusing the slopes already computed for the layers above, so the whole gradient costs only about twice a forward pass. The same rule explains a classic problem: multiplying many slopes smaller than 1 makes the product vanish, which is why deep recurrent networks struggled with long-range dependencies and why residual connections, which add a path with slope 1, help deep transformers train.
Easy to confuse
- The derivative is a rate, not a change: a slope of 24 means y changes about 24 times as much as a small change in x.
- A zero slope means a flat spot. It can be a minimum, a maximum or a plateau.
The derivative of a loss with respect to a weight is positive. To lower the loss, the weight should…
A positive slope means the loss rises as the weight rises. Moving the weight the other way (against the gradient) lowers it. That is gradient descent.
For y = (3x + 1)², what is dy/dx at x = 0?
Chain rule: let u = 3x + 1, y = u². dy/dx = 2u · 3 = 6u. At x = 0, u = 1, so the slope is 6.
What does the chain rule let backpropagation do?
A network is a chain of simple functions. The chain rule says the overall slope is the product of each step’s slope, so gradients can be passed backward layer by layer.
Read more in the books
- Deep Learning, Goodfellow, Bengio, Courville: Ch. 4, “Numerical Computation”: gradient-based optimization.
- Mathematics for Machine Learning, Deisenroth, Faisal, Ong: Ch. 5, “Vector Calculus”.
- Dive into Deep Learning, Zhang, Lipton, Li, Smola: Ch. 2, “Preliminaries”: calculus.