LLM Lab

Classification

Classification puts each input into a class. A linear classifier draws a boundary, and a sigmoid turns distance from it into a probability.

Where this sits in the training loop
Data→Model makes a prediction→Loss measures the error→Gradients (backprop)→Update the weights (optimiser)

A boundary and a probability

Logistic regression computes a score z = w₁x₁ + w₂x₂ + b. The boundary is where z = 0; the sigmoid σ(z) gives the probability of the orange class, rising as points sit further on its side.

0.050.270.620.92score (signed distance to the boundary) →
The sigmoid turns a point’s signed distance from the boundary into a probability: 0.5 on the line, near 1 far on one side, near 0 far on the other.

Accuracy vs cross-entropy

Accuracy counts the points on the correct side. Training uses cross-entropy instead, −log p(true class), averaged over points: it is smooth enough for gradient descent and punishes confident mistakes hard. Rotate and shift the boundary in the lab and watch both numbers.

The same loss trains language models: predicting the next token is classification over the whole vocabulary.

A worked example

A point at signed distance 2.5 from the boundary on its correct side gets probability σ(2.5) ≈ 0.92 and costs −log 0.92 ≈ 0.08. A point just over the line on the wrong side, at −0.5, gets 0.38 for its true class and costs about 0.97. Both kinds of point change the loss, but only the second changes the accuracy, which is why training uses the loss, not accuracy.

In a language model

Predicting the next token is classification with tens of thousands of classes. Softmax plays the role the sigmoid plays here, and the same cross-entropy loss is used. A fine-tuned classifier, such as a sentiment model built on BERT, is a single linear layer and softmax on top of the transformer’s output vector.

Easy to confuse

Check yourself · 1 of 3

Logistic regression turns a score into a probability with…

Check yourself · 2 of 3

Two boundaries both get 100% accuracy. Cross-entropy can still prefer one because…

Check yourself · 3 of 3

An LLM choosing the next token is a classifier over…

Read more in the books

  • Dive into Deep Learning, Zhang, Lipton, Li, Smola: Ch. 4, “Linear Neural Networks for Classification”.
  • Pattern Recognition and Machine Learning, Christopher Bishop: Ch. 4, “Linear Models for Classification”.
  • Hands-On Machine Learning with Scikit-Learn, Keras and TensorFlow (3rd ed.), Aurélien Géron: Ch. 3, “Classification”.
All books
Labmove the boundary
How to read this
  • Two classes of points (two colours) and a straight boundary. Drag the angle and offset to move the boundary.
  • The readout reports accuracy (points on the correct side) and the cross-entropy loss, which also counts how confidently each point is classified. Misclassified points have a ring.