Data & features
A model only sees numbers in columns. What you put in those columns decides what it can learn.
Rows, columns, target
A table has one row per example and one column per feature. One column is special: the target, the value you want to predict. Training means finding a function from the features to the target that works on the rows you have.
Good features, bad features
A useful feature carries information about the target: the size of a house says something about its price. Other columns look like features but are not: a row ID is just a counter, and a zip code stored as a number has an arbitrary order. Adding them gives the model more ways to fit noise.
Keep rows back for testing
Split the rows into a training set and a test set. Fit on the first, report the error on the second. The lab does this for you: tick the columns the model may use and watch both errors.
A worked example
With only the size column, the model’s error is about 20 on training rows and 24 on test rows: size explains most of the price. Adding rooms barely changes the test error, since rooms mostly repeat what size says. Adding the row ID and the zip code lowers the training error to about 16 but raises the test error to about 37: the model used the noise to fit the training rows and does worse on new ones.
In a language model
For an LLM the rows are documents, the features are tokens, and the target is the next token, so there are no hand-made columns. The same discipline still applies: evaluate on text the model has not trained on, and watch for leaks such as test questions that appear in the training data.
Easy to confuse
- A lower training error does not mean a better model; only the held-out error tells you how it will do on new data.
- The test set is for the final check. Choosing settings by looking at it again and again quietly turns it into training data.
A column of row IDs (1, 2, 3, …) is added as a feature. What do you expect?
An ID carries no information about the target. With enough capacity a model can use it to memorise rows, which looks good on training data and fails on new data.
Why is the model scored on rows it never trained on?
A model’s job is to predict new cases. The held-out rows stand in for them, so the held-out error is the number that matters.
A zip code is stored as a number (10110, 10250, …). Why is it risky as a raw numeric feature?
A linear model treats the value as a quantity. Categories like zip codes usually need a different encoding, such as one column per code.
Read more in the books
- Hands-On Machine Learning with Scikit-Learn, Keras and TensorFlow (3rd ed.), Aurélien Géron: Ch. 2, “End-to-End Machine Learning Project”.
- Designing Machine Learning Systems, Chip Huyen: the chapters on training data and feature engineering.
- Deep Learning, Goodfellow, Bengio, Courville: Ch. 5, “Machine Learning Basics”.