LLM Lab

LLM-as-judge

Open-ended answers have no single reference. A common shortcut is to ask a strong model to judge them, which works until its own biases leak into the scores.

Where this sits when evaluating a model
Choose test data→Score the outputs→Compare models→Check trust (calibration, leaks)→Evaluate whole systems→Measure speed and cost

Pairwise judging

Show the judge a question and two answers, A and B, and ask which is better. Repeat over many questions to get a win rate. It scales far better than human review, and on many tasks it agrees with people fairly often.

Position bias

Judges tend to favour the answer in a particular position, often the first. If model A is always shown first, part of its “win rate” is just the slot it sat in.

A firstverdict: A
B firstverdict: B (the first one again)
resultinconsistent → count as a tie
Swapping the order exposes position bias: a judge that picks whichever answer comes first gives opposite verdicts. Judging both orders and keeping only consistent wins removes most of the effect.

The lab simulates a judge with an adjustable position bias. The two models are truly equal unless you change the quality gap. Compare judging in one order with judging in both orders.

A worked example

Suppose A and B are truly equal but the judge favours the first position by 15 points. With A always shown first, A appears to win about 65% of the time. Judging each pair twice, A first and then B first, and counting only consistent verdicts as wins, brings the estimate back near 50%.

In practice

Strong models used as judges agree with people about as often as people agree with each other on many tasks, which makes them useful for fast iteration. Known biases include position, length (longer answers look better) and self-preference (a judge favouring its own model’s style). Write a clear rubric, swap positions, and check a sample of verdicts by hand.

Easy to confuse

Check yourself · 1 of 3

A judge model prefers whichever answer it sees first. How do you control for that?

Check yourself · 2 of 3

Which of these is a known bias of LLM judges?

Check yourself · 3 of 3

What should you check before trusting an LLM judge on a new task?

Read more in the books

  • AI Engineering, Chip Huyen: Ch. 3, “Evaluation Methodology”: AI as a judge.
  • AI Engineering, Chip Huyen: Ch. 4, “Evaluate AI Systems”: design your evaluation pipeline.
All books
Labposition-bias demo
How to read this
  • Set how much better answer A truly is than B, and how strongly the judge prefers whichever answer it sees first.
  • Judge 500 pairs with A always first, or judge both orders; the result compares the win rate the judge reports with the true one.
Run a judging round.