LLM Lab

Human eval & arenas

Some qualities are easiest to judge side by side. Arenas show people two anonymous answers, ask which is better, and turn thousands of votes into a ranking.

Where this sits when evaluating a model
Choose test data→Score the outputs→Compare models→Check trust (calibration, leaks)→Evaluate whole systems→Measure speed and cost

From votes to ratings

The Elo system gives each model a rating. Before a battle it predicts the winner’s chance from the rating gap: E = 1 / (1 + 10^((Rb − Ra)/400)). After the battle, each rating moves by K × (result − E): an upset moves ratings a lot, an expected win barely.

Noise and order

Ratings start equal and drift toward the truth as votes come in. Run the lab: three simulated models with hidden true strengths fight random battles. Watch how many battles it takes for the order to settle, and how a large K makes the ratings jumpy. Many leaderboards fit all votes at once (a Bradley–Terry model) instead of updating one by one, which removes the dependence on vote order.

A worked example

Model A is rated 1500 and B 1600. The Elo formula expects A to win 1 / (1 + 10^((1600 − 1500)/400)) ≈ 36% of the time. If A wins anyway, with K = 32 it gains 32 × (1 − 0.36) ≈ 20.5 points and B loses the same. Had B won, as expected, only about 11.5 points would have moved.

expectedA (1500) vs B (1600): E = 1 / (1 + 10^(100/400)) = 0.36
A winsA: 1500 + 32 × (1 − 0.36) = 1520.5
B: 1600 − 20.5 = 1579.5
One Elo update with K = 32. The lower-rated model was expected to win 36% of the time; its upset win moves 20.5 points from B to A. An expected result moves far fewer points.

In practice

Public arenas collect many thousands of votes from people who compare two anonymous answers to their own prompt. Ratings are usually fitted over all votes at once (a Bradley–Terry model) rather than updated one by one, which removes the dependence on the order of battles, and reported with confidence intervals.

Easy to confuse

Check yourself · 1 of 3

A model rated 1600 meets one rated 1400. What does Elo expect?

Check yourself · 2 of 3

Raising the K-factor makes ratings…

Check yourself · 3 of 3

Why do arenas collect many votes on varied prompts?

Read more in the books

  • AI Engineering, Chip Huyen: Ch. 3, “Evaluation Methodology”: ranking models with comparative evaluation.
  • AI Engineering, Chip Huyen: Ch. 4, “Evaluate AI Systems”: model selection.
All books
LabElo simulator
How to read this
  • Each model has a true strength (hidden) and an Elo rating that starts equal. Run 100 battles to simulate votes; the chart tracks the ratings.
  • The K-factor sets how much one battle moves a rating: high K reacts fast but stays noisy.