Neural nets & backprop
A neural network is a chain of simple operations. Backpropagation finds how much each weight is to blame for the loss, all in one backward sweep.
Forward, then backward
The forward pass computes each node from its inputs, ending at the loss. The backward pass starts from ∂L/∂L = 1 and moves back one node at a time, multiplying by each operation’s local derivative: the chain rule.
The lab is a one-neuron network: a = ReLU(x·w + b), with loss (a − y)². Step through both passes, predicting each gradient, then apply one gradient-descent update.
Why it scales
Every operation only needs its local derivative. Frameworks record the graph during the forward pass and replay it backwards, so the same procedure trains a network with billions of weights.
A worked example
Take one weight w = 1.5, a bias b = 0.5, an input x = 2 and a target y = 4. Forward: the prediction is 1.5 × 2 + 0.5 = 3.5, the error −0.5 and the squared-error loss 0.25. Backward: the loss changes at rate 2 × (−0.5) = −1 with the prediction, so dL/dw = −1 × x = −2 and dL/db = −1. Gradient descent with learning rate 0.1 then moves w to 1.7 and b to 0.6, and the loss drops.
In a language model
The same bookkeeping runs over billions of weights. Frameworks such as PyTorch record every operation of the forward pass and replay it backwards automatically, which is called automatic differentiation. Storing the forward values for the backward pass is a big part of the memory used in training.
Easy to confuse
- Backpropagation computes the gradient; the optimiser (SGD, Adam) decides how to use it.
- The gradient of a weight depends on the input that flowed through it: a weight on an input of zero gets no gradient from that example.
Backpropagation computes…
It walks the computation graph backwards, multiplying local derivatives, so one pass gives every gradient.
Why does backprop go from the loss towards the inputs?
By the chain rule, ∂L/∂node = ∂L/∂next · ∂next/∂node, so later gradients must be known first.
A ReLU’s input is negative. What gradient flows through it?
ReLU is flat for negative inputs, so its derivative there is 0 and nothing upstream learns from this example.
Read more in the books
- Deep Learning, Goodfellow, Bengio, Courville: Ch. 6, “Deep Feedforward Networks”, section 6.5 (back-propagation).
- Dive into Deep Learning, Zhang, Lipton, Li, Smola: Ch. 5, “Multilayer Perceptrons”, forward and backward propagation and computational graphs.
- Build a Large Language Model (From Scratch), Sebastian Raschka: Appendix A, “Introduction to PyTorch” (automatic differentiation).