LLM Lab

Neural nets & backprop

A neural network is a chain of simple operations. Backpropagation finds how much each weight is to blame for the loss, all in one backward sweep.

Where this sits in the training loop
Data→Model makes a prediction→Loss measures the error→Gradients (backprop)→Update the weights (optimiser)

Forward, then backward

The forward pass computes each node from its inputs, ending at the loss. The backward pass starts from ∂L/∂L = 1 and moves back one node at a time, multiplying by each operation’s local derivative: the chain rule.

The lab is a one-neuron network: a = ReLU(x·w + b), with loss (a − y)². Step through both passes, predicting each gradient, then apply one gradient-descent update.

Why it scales

Every operation only needs its local derivative. Frameworks record the graph during the forward pass and replay it backwards, so the same procedure trains a network with billions of weights.

A worked example

Take one weight w = 1.5, a bias b = 0.5, an input x = 2 and a target y = 4. Forward: the prediction is 1.5 × 2 + 0.5 = 3.5, the error −0.5 and the squared-error loss 0.25. Backward: the loss changes at rate 2 × (−0.5) = −1 with the prediction, so dL/dw = −1 × x = −2 and dL/db = −1. Gradient descent with learning rate 0.1 then moves w to 1.7 and b to 0.6, and the loss drops.

w1.5grad −2× x = 2 →w·x + b3.5grad −1− y = 4 →error−0.5grad −1² →loss0.25grad 1
Forward (black): with x = 2, w = 1.5, b = 0.5 and target y = 4, the prediction is 3.5 and the loss 0.25. Backward (orange): the loss’s slope is passed back step by step; dL/dw = −1 × x = −2, so increasing w lowers the loss.

In a language model

The same bookkeeping runs over billions of weights. Frameworks such as PyTorch record every operation of the forward pass and replay it backwards automatically, which is called automatic differentiation. Storing the forward values for the backward pass is a big part of the memory used in training.

Easy to confuse

Check yourself · 1 of 3

Backpropagation computes…

Check yourself · 2 of 3

Why does backprop go from the loss towards the inputs?

Check yourself · 3 of 3

A ReLU’s input is negative. What gradient flows through it?

Read more in the books

  • Deep Learning, Goodfellow, Bengio, Courville: Ch. 6, “Deep Feedforward Networks”, section 6.5 (back-propagation).
  • Dive into Deep Learning, Zhang, Lipton, Li, Smola: Ch. 5, “Multilayer Perceptrons”, forward and backward propagation and computational graphs.
  • Build a Large Language Model (From Scratch), Sebastian Raschka: Appendix A, “Introduction to PyTorch” (automatic differentiation).
All books
Labstep the graph
How to read this
  • The graph shows a tiny network as boxes; each step of the walkthrough highlights the box being computed.
  • Forward steps fill in values from left to right; backward steps fill in gradients from the loss back to the weights.
  • Some steps ask you to predict the next number before it is shown; Back and Next move through the steps.