LLM Lab

Hallucination & calibration

Being right is one question. Knowing when you might be wrong is another, and it decides whether a model’s confidence can be trusted.

Where this sits when evaluating a model
Choose test data→Score the outputs→Compare models→Check trust (calibration, leaks)→Evaluate whole systems→Measure speed and cost

Reliability diagrams

Group answers by the model’s confidence: all the 60–70% answers together, all the 70–80% ones, and so on. For a calibrated model, accuracy in each bin matches its confidence, and the bars sit on the diagonal. Bars below the diagonal mean overconfidence.

Expected calibration error (ECE) averages the gap between confidence and accuracy over the bins, weighted by how many answers fall in each.

Fixing it

The simulated model in the lab is overconfident. Temperature scaling divides its logits by T before the softmax; fitted on held-out data, a single number often removes most of the miscalibration. Find the T that brings ECE down.

A worked example

Collect 1,000 answers where the model said it was 95% sure. If 740 of them are right, that group sits at 74% on the diagram, far below the diagonal: the model is overconfident. ECE averages such gaps over all groups, weighted by how many answers each holds. Dividing the logits by a temperature above 1 softens the confidences and often closes most of the gap.

said 60% right 58%
said 80% right 71%
said 95% right 74%
Grouping answers by stated confidence. When the model says 95% it is right only 74% of the time: overconfident. A calibrated model’s teal bars would match the grey ones.

In practice

Calibration matters when answers feed decisions: a well-calibrated model can say “I am not sure” and be believed. Pretrained models are often well calibrated on multiple-choice questions, and preference tuning can make them more overconfident. Hallucinations are the extreme case: fluent, confident answers with nothing behind them.

Easy to confuse

Check yourself · 1 of 3

A model says “90% sure” on 100 answers and gets 70 right. It is…

Check yourself · 2 of 3

Temperature scaling changes which answer the model picks.

Check yourself · 3 of 3

How does calibration relate to hallucination?

Read more in the books

  • AI Engineering, Chip Huyen: Ch. 4, “Evaluate AI Systems”: evaluation criteria (factual consistency).
  • AI Engineering, Chip Huyen: Ch. 3, “Evaluation Methodology”: understanding language modeling metrics.
All books
Labreliability diagram
How to read this
  • The reliability diagram groups answers by the confidence the model gave. Each bar is how often answers in that group were right; the diagonal is perfect calibration.
  • The temperature slider rescales the model’s confidence. The readout reports the expected calibration error (ECE): lower is better calibrated.