Classification
Classification puts each input into a class. A linear classifier draws a boundary, and a sigmoid turns distance from it into a probability.
A boundary and a probability
Logistic regression computes a score z = w₁x₁ + w₂x₂ + b. The boundary is where z = 0; the sigmoid σ(z) gives the probability of the orange class, rising as points sit further on its side.
Accuracy vs cross-entropy
Accuracy counts the points on the correct side. Training uses cross-entropy instead, −log p(true class), averaged over points: it is smooth enough for gradient descent and punishes confident mistakes hard. Rotate and shift the boundary in the lab and watch both numbers.
The same loss trains language models: predicting the next token is classification over the whole vocabulary.
A worked example
A point at signed distance 2.5 from the boundary on its correct side gets probability σ(2.5) ≈ 0.92 and costs −log 0.92 ≈ 0.08. A point just over the line on the wrong side, at −0.5, gets 0.38 for its true class and costs about 0.97. Both kinds of point change the loss, but only the second changes the accuracy, which is why training uses the loss, not accuracy.
In a language model
Predicting the next token is classification with tens of thousands of classes. Softmax plays the role the sigmoid plays here, and the same cross-entropy loss is used. A fine-tuned classifier, such as a sentiment model built on BERT, is a single linear layer and softmax on top of the transformer’s output vector.
Easy to confuse
- A linear boundary can only separate classes that a straight line (or plane) can split; neural networks bend the space first so that a line works.
- High accuracy with a high loss means the model is right but unsure, or very wrong on a few points.
Logistic regression turns a score into a probability with…
σ(z) = 1 / (1 + e^(−z)) squashes any real score into (0, 1).
Two boundaries both get 100% accuracy. Cross-entropy can still prefer one because…
Accuracy only checks which side a point is on; cross-entropy also looks at how far, through the predicted probability.
An LLM choosing the next token is a classifier over…
Next-token prediction is classification with tens of thousands of classes, trained with the same cross-entropy loss.
Read more in the books
- Dive into Deep Learning, Zhang, Lipton, Li, Smola: Ch. 4, “Linear Neural Networks for Classification”.
- Pattern Recognition and Machine Learning, Christopher Bishop: Ch. 4, “Linear Models for Classification”.
- Hands-On Machine Learning with Scikit-Learn, Keras and TensorFlow (3rd ed.), Aurélien Géron: Ch. 3, “Classification”.