LLM Lab

Linear regression

The simplest model: a straight line. Fitting it already contains the whole recipe of machine learning: a model, a loss, and a search for the best parameters.

Where this sits in the training loop
Data→Model makes a prediction→Loss measures the error→Gradients (backprop)→Update the weights (optimiser)

A model with two numbers

The model predicts ŷ = a·x + b. The slope a and intercept b are its parameters: everything it learns lives in them.

A loss to compare lines

For each point, the error is the vertical gap between the prediction and the truth. The mean squared error averages the squared gaps. Lower is better, and the best line is the one with the smallest MSE.

Each dashed segment is one error: the gap between a point and the line. The mean squared error squares these gaps and averages them; fitting means choosing the slope and intercept that make it smallest.

Move the sliders to fit the line by hand, then press Fit to jump to the least-squares solution. Large language models are fitted the same way, with billions of parameters instead of two.

A worked example

Take the line ŷ = 1.35x + 0.9 and the point (2, 3.9). The prediction is 1.35 × 2 + 0.9 = 3.6, so the error is −0.3 and the squared error 0.09. Doing this for every point and averaging gives the MSE. The least-squares fit solves for the slope and intercept that minimise it in one step; gradient descent reaches the same answer step by step, which is how larger models are trained.

In a language model

Every linear layer inside a transformer is the same idea with many inputs and outputs: y = W·x + b, where W has millions of entries instead of one slope. The training loss is cross-entropy rather than squared error, but the recipe is unchanged: a model with parameters, a number that measures how wrong it is, and a way to adjust the parameters to lower it.

Easy to confuse

Check yourself · 1 of 3

Why square the errors instead of just adding them?

Check yourself · 2 of 3

Doubling one point’s distance from the line multiplies its contribution to MSE by…

Check yourself · 3 of 3

What does “fitting” the line mean here?

Read more in the books

  • Deep Learning, Goodfellow, Bengio, Courville: Ch. 5, “Machine Learning Basics”, section 5.1.4 (linear regression).
  • Dive into Deep Learning, Zhang, Lipton, Li, Smola: Ch. 3, “Linear Neural Networks for Regression”.
  • Pattern Recognition and Machine Learning, Christopher Bishop: Ch. 3, “Linear Models for Regression”.
All books
Labfit the line
How to read this
  • Black dots are the data; the teal line is the model ŷ = a·x + b. Drag the slope a and intercept b, or press Fit to jump to the best line.
  • The strip under the readout shows each point’s error ŷ − y: teal where the line is above the point, orange where it is below.
  • Squaring and averaging those errors gives the MSE in the readout; the best line makes it as small as possible.