Precision, recall & F1
When an LLM labels things (spam or not, safe or unsafe, right answer or wrong), accuracy alone can hide what really goes wrong.
Four kinds of outcome
For a yes/no task each prediction is a true positive (said yes, was yes), false positive (said yes, was no), false negative (said no, was yes) or true negative. The confusion matrix counts all four.
Precision, recall and F1
Precision asks: of the items flagged, how many were right? Recall asks: of the items that should be flagged, how many were found? Raising the threshold usually raises precision and lowers recall. F1 is their harmonic mean, which is high only when both are. For several classes, compute the metric per class and average: macro averaging weights every class equally, micro averaging weights every item equally.
A worked example
A moderation model scores 100 messages, 20 of them truly harmful. At threshold 0.5 it flags 10, of which 8 are harmful. Precision is 0.80 and recall 0.40: it rarely cries wolf but misses most harmful messages. Lowering the threshold might flag 25 with 16 harmful: precision 0.64, recall 0.80. Which is better depends on the cost of each mistake, which is a product decision, not a metric.
In practice
These metrics apply whenever an LLM acts as a classifier: intent detection, content filters, LLM-as-judge verdicts compared with human labels, and retrieval relevance. Report precision and recall, not only accuracy, and choose the threshold on a validation set, not the test set.
Easy to confuse
- Precision and recall swap roles if you swap which class you call “positive”.
- High accuracy on an imbalanced dataset says little; check the rare class.
A spam filter flags 10 emails; 8 are really spam. What is its precision?
Precision is the share of flagged items that are truly positive: 8 / 10. The missed spam affects recall, not precision.
There are 20 spam emails in total and the filter catches 8. What is its recall?
Recall is the share of all true positives that were found: 8 / 20 = 0.4.
Why use F1 instead of accuracy when only 2% of items are positive?
With rare positives, accuracy rewards doing nothing. F1 combines precision and recall on the positive class, so it drops to 0 for a model that never finds a positive.
Read more in the books
- Hands-On Machine Learning with Scikit-Learn, Keras and TensorFlow (3rd ed.), Aurélien Géron: Ch. 3, “Classification”: precision, recall, the precision/recall trade-off.
- Introduction to Information Retrieval, Manning, Raghavan, Schütze: Ch. 8, “Evaluation in information retrieval”.
- AI Engineering, Chip Huyen: Ch. 4, “Evaluate AI Systems”.