LLM Lab

Precision, recall & F1

When an LLM labels things (spam or not, safe or unsafe, right answer or wrong), accuracy alone can hide what really goes wrong.

Where this sits when evaluating a model
Choose test data→Score the outputs→Compare models→Check trust (calibration, leaks)→Evaluate whole systems→Measure speed and cost

Four kinds of outcome

For a yes/no task each prediction is a true positive (said yes, was yes), false positive (said yes, was no), false negative (said no, was yes) or true negative. The confusion matrix counts all four.

spamnot spamflagged82not flagged1278
A confusion matrix for 100 emails. Precision = 8 / (8 + 2) = 0.80, recall = 8 / (8 + 12) = 0.40, F1 = 2 × 0.8 × 0.4 / (0.8 + 0.4) ≈ 0.53, while accuracy is (8 + 78) / 100 = 0.86.

Precision, recall and F1

Precision asks: of the items flagged, how many were right? Recall asks: of the items that should be flagged, how many were found? Raising the threshold usually raises precision and lowers recall. F1 is their harmonic mean, which is high only when both are. For several classes, compute the metric per class and average: macro averaging weights every class equally, micro averaging weights every item equally.

A worked example

A moderation model scores 100 messages, 20 of them truly harmful. At threshold 0.5 it flags 10, of which 8 are harmful. Precision is 0.80 and recall 0.40: it rarely cries wolf but misses most harmful messages. Lowering the threshold might flag 25 with 16 harmful: precision 0.64, recall 0.80. Which is better depends on the cost of each mistake, which is a product decision, not a metric.

In practice

These metrics apply whenever an LLM acts as a classifier: intent detection, content filters, LLM-as-judge verdicts compared with human labels, and retrieval relevance. Report precision and recall, not only accuracy, and choose the threshold on a validation set, not the test set.

Easy to confuse

Check yourself · 1 of 3

A spam filter flags 10 emails; 8 are really spam. What is its precision?

Check yourself · 2 of 3

There are 20 spam emails in total and the filter catches 8. What is its recall?

Check yourself · 3 of 3

Why use F1 instead of accuracy when only 2% of items are positive?

Read more in the books

  • Hands-On Machine Learning with Scikit-Learn, Keras and TensorFlow (3rd ed.), Aurélien Géron: Ch. 3, “Classification”: precision, recall, the precision/recall trade-off.
  • Introduction to Information Retrieval, Manning, Raghavan, Schütze: Ch. 8, “Evaluation in information retrieval”.
  • AI Engineering, Chip Huyen: Ch. 4, “Evaluate AI Systems”.
All books
Labmove the threshold
How to read this
  • Each dot is one item with the model’s score from 0 to 1; filled dots are truly positive. Items to the right of the threshold line are predicted positive.
  • Drag the threshold. The confusion matrix counts true and false positives and negatives; the tiles turn those counts into precision, recall, F1 and accuracy.
  • The curve plots precision against recall for every threshold; the orange dot is where your threshold sits.