LLM Lab

Data & features

A model only sees numbers in columns. What you put in those columns decides what it can learn.

Where this sits in the training loop
Data→Model makes a prediction→Loss measures the error→Gradients (backprop)→Update the weights (optimiser)

Rows, columns, target

A table has one row per example and one column per feature. One column is special: the target, the value you want to predict. Training means finding a function from the features to the target that works on the rows you have.

sizeroomsagepricetrain95312213train140430299train62241103test11835287test77222162
Rows are examples, columns are features, and one column is the target (price, outlined). Some rows are held back as a test set: the model never trains on them, so its error there estimates how it does on new data.

Good features, bad features

A useful feature carries information about the target: the size of a house says something about its price. Other columns look like features but are not: a row ID is just a counter, and a zip code stored as a number has an arbitrary order. Adding them gives the model more ways to fit noise.

Keep rows back for testing

Split the rows into a training set and a test set. Fit on the first, report the error on the second. The lab does this for you: tick the columns the model may use and watch both errors.

A worked example

With only the size column, the model’s error is about 20 on training rows and 24 on test rows: size explains most of the price. Adding rooms barely changes the test error, since rooms mostly repeat what size says. Adding the row ID and the zip code lowers the training error to about 16 but raises the test error to about 37: the model used the noise to fit the training rows and does worse on new ones.

In a language model

For an LLM the rows are documents, the features are tokens, and the target is the next token, so there are no hand-made columns. The same discipline still applies: evaluate on text the model has not trained on, and watch for leaks such as test questions that appear in the training data.

Easy to confuse

Check yourself · 1 of 3

A column of row IDs (1, 2, 3, …) is added as a feature. What do you expect?

Check yourself · 2 of 3

Why is the model scored on rows it never trained on?

Check yourself · 3 of 3

A zip code is stored as a number (10110, 10250, …). Why is it risky as a raw numeric feature?

Read more in the books

  • Hands-On Machine Learning with Scikit-Learn, Keras and TensorFlow (3rd ed.), Aurélien Géron: Ch. 2, “End-to-End Machine Learning Project”.
  • Designing Machine Learning Systems, Chip Huyen: the chapters on training data and feature engineering.
  • Deep Learning, Goodfellow, Bengio, Courville: Ch. 5, “Machine Learning Basics”.
All books
Labsort the table
How to read this
  • The table shows the first 6 of 18 houses; the last column, price, is the target. Rows 1 to 12 train the model, rows 13 to 18 test it.
  • Tick the columns the model may use. A linear model is refitted on the training rows each time.
  • The two tiles compare the error on the training rows with the error on the test rows (lower is better); the note flags when the gap grows.
Columns the model may use
Training error–
Test error–