Linear regression
The simplest model: a straight line. Fitting it already contains the whole recipe of machine learning: a model, a loss, and a search for the best parameters.
A model with two numbers
The model predicts ŷ = a·x + b. The slope a and intercept b are its parameters: everything it learns lives in them.
A loss to compare lines
For each point, the error is the vertical gap between the prediction and the truth. The mean squared error averages the squared gaps. Lower is better, and the best line is the one with the smallest MSE.
Move the sliders to fit the line by hand, then press Fit to jump to the least-squares solution. Large language models are fitted the same way, with billions of parameters instead of two.
A worked example
Take the line ŷ = 1.35x + 0.9 and the point (2, 3.9). The prediction is 1.35 × 2 + 0.9 = 3.6, so the error is −0.3 and the squared error 0.09. Doing this for every point and averaging gives the MSE. The least-squares fit solves for the slope and intercept that minimise it in one step; gradient descent reaches the same answer step by step, which is how larger models are trained.
In a language model
Every linear layer inside a transformer is the same idea with many inputs and outputs: y = W·x + b, where W has millions of entries instead of one slope. The training loss is cross-entropy rather than squared error, but the recipe is unchanged: a model with parameters, a number that measures how wrong it is, and a way to adjust the parameters to lower it.
Easy to confuse
- “Linear” refers to the parameters: a model can use x² as an input and still be linear regression.
- Squaring the errors punishes large mistakes much more than small ones, so a single outlier can pull the line noticeably.
Why square the errors instead of just adding them?
A point 2 above and a point 2 below would sum to zero error. Squaring makes every error count and punishes big misses more.
Doubling one point’s distance from the line multiplies its contribution to MSE by…
The error is squared, so twice the distance gives four times the contribution. One outlier can pull the line a long way.
What does “fitting” the line mean here?
Training picks the parameters with the lowest loss on the data; with noise, no line passes through every point.
Read more in the books
- Deep Learning, Goodfellow, Bengio, Courville: Ch. 5, “Machine Learning Basics”, section 5.1.4 (linear regression).
- Dive into Deep Learning, Zhang, Lipton, Li, Smola: Ch. 3, “Linear Neural Networks for Regression”.
- Pattern Recognition and Machine Learning, Christopher Bishop: Ch. 3, “Linear Models for Regression”.