Hallucination & calibration
Being right is one question. Knowing when you might be wrong is another, and it decides whether a model’s confidence can be trusted.
Reliability diagrams
Group answers by the model’s confidence: all the 60–70% answers together, all the 70–80% ones, and so on. For a calibrated model, accuracy in each bin matches its confidence, and the bars sit on the diagonal. Bars below the diagonal mean overconfidence.
Expected calibration error (ECE) averages the gap between confidence and accuracy over the bins, weighted by how many answers fall in each.
Fixing it
The simulated model in the lab is overconfident. Temperature scaling divides its logits by T before the softmax; fitted on held-out data, a single number often removes most of the miscalibration. Find the T that brings ECE down.
A worked example
Collect 1,000 answers where the model said it was 95% sure. If 740 of them are right, that group sits at 74% on the diagram, far below the diagonal: the model is overconfident. ECE averages such gaps over all groups, weighted by how many answers each holds. Dividing the logits by a temperature above 1 softens the confidences and often closes most of the gap.
In practice
Calibration matters when answers feed decisions: a well-calibrated model can say “I am not sure” and be believed. Pretrained models are often well calibrated on multiple-choice questions, and preference tuning can make them more overconfident. Hallucinations are the extreme case: fluent, confident answers with nothing behind them.
Easy to confuse
- Calibration and accuracy are different: a model can be accurate and badly calibrated, or calibrated and often wrong.
- The confidence that matters is the model’s probability for its answer, not how confident its wording sounds.
A model says “90% sure” on 100 answers and gets 70 right. It is…
Calibrated means stated confidence matches accuracy. Here confidence (90%) exceeds accuracy (70%).
Temperature scaling changes which answer the model picks.
Dividing all logits by the same T keeps their order, so the top answer is unchanged; only the probabilities shrink or stretch.
How does calibration relate to hallucination?
If low confidence reliably signals errors, a system can refuse or ask for help instead of stating a wrong answer.
Read more in the books
- AI Engineering, Chip Huyen: Ch. 4, “Evaluate AI Systems”: evaluation criteria (factual consistency).
- AI Engineering, Chip Huyen: Ch. 3, “Evaluation Methodology”: understanding language modeling metrics.