Human eval & arenas
Some qualities are easiest to judge side by side. Arenas show people two anonymous answers, ask which is better, and turn thousands of votes into a ranking.
From votes to ratings
The Elo system gives each model a rating. Before a battle it predicts the winner’s chance from the rating gap: E = 1 / (1 + 10^((Rb − Ra)/400)). After the battle, each rating moves by K × (result − E): an upset moves ratings a lot, an expected win barely.
Noise and order
Ratings start equal and drift toward the truth as votes come in. Run the lab: three simulated models with hidden true strengths fight random battles. Watch how many battles it takes for the order to settle, and how a large K makes the ratings jumpy. Many leaderboards fit all votes at once (a Bradley–Terry model) instead of updating one by one, which removes the dependence on vote order.
A worked example
Model A is rated 1500 and B 1600. The Elo formula expects A to win 1 / (1 + 10^((1600 − 1500)/400)) ≈ 36% of the time. If A wins anyway, with K = 32 it gains 32 × (1 − 0.36) ≈ 20.5 points and B loses the same. Had B won, as expected, only about 11.5 points would have moved.
In practice
Public arenas collect many thousands of votes from people who compare two anonymous answers to their own prompt. Ratings are usually fitted over all votes at once (a Bradley–Terry model) rather than updated one by one, which removes the dependence on the order of battles, and reported with confidence intervals.
Easy to confuse
- A rating difference translates into a win probability, not a percentage of quality: 100 points means about a 64% win rate.
- Arena votes reflect the prompts and preferences of the people voting, which may differ from your use case.
A model rated 1600 meets one rated 1400. What does Elo expect?
Expected score = 1 / (1 + 10^(−200/400)) ≈ 0.76.
Raising the K-factor makes ratings…
K is the step size of each update: large K chases recent results, small K averages over many.
Why do arenas collect many votes on varied prompts?
Ratings are only as good as the volume and mix of comparisons behind them; confidence intervals shrink with more votes.
Read more in the books
- AI Engineering, Chip Huyen: Ch. 3, “Evaluation Methodology”: ranking models with comparative evaluation.
- AI Engineering, Chip Huyen: Ch. 4, “Evaluate AI Systems”: model selection.