RLHF & DPO
A pretrained model can continue text, but not necessarily the way people want. Preference tuning teaches it which of two answers people prefer.
Preference data
For each prompt, show people two model answers and record which is better. That is the raw material: thousands of (prompt, chosen, rejected) triples. Try labelling a few in the lab.
RLHF and DPO
RLHF first trains a reward model to predict the preferences, then uses reinforcement learning to make the language model produce high-reward answers, with a penalty for drifting too far from the original model.
DPO (direct preference optimisation) gets the same effect with one supervised-style loss: raise the probability of the chosen answer relative to the rejected one, measured against a frozen reference copy. The β setting controls how far the model may move.
A worked example
For the prompt “Explain what a KV cache is in one sentence”, a labeller prefers the precise answer over the vague one. RLHF first trains a reward model to score the preferred answer higher across many such pairs, then optimises the language model with reinforcement learning to earn high reward while staying close to its starting point. DPO skips the reward model: its loss directly raises the probability of the chosen answer relative to the rejected one, compared with a frozen reference model.
In practice
Preference tuning follows supervised fine-tuning. Llama 3, for example, used supervised fine-tuning, rejection sampling and DPO. Preference data can come from people or from another model acting as a judge, and its quality largely decides how helpful and how safe the final model is.
Easy to confuse
- The reward model is a separate network used only during RLHF training; it is not part of the deployed model.
- Optimising too hard for a reward model can exploit its blind spots (reward hacking), which is why both methods keep the model close to a reference.
What do RLHF and DPO learn from?
Both use preference data: for a prompt, a chosen and a rejected response.
What does DPO skip compared with classic RLHF?
DPO turns the preference objective into a classification-style loss on the policy directly.
Why keep the tuned model close to the original (the KL term / β)?
Optimising hard against an imperfect reward signal leads to reward hacking; the constraint keeps changes modest.
Read more in the books
- Reinforcement Learning from Human Feedback, Nathan Lambert: the chapters on reward modeling and direct alignment.
- AI Engineering, Chip Huyen: Ch. 2, the section on post-training (supervised finetuning and preference finetuning).
- Hands-On Large Language Models, Jay Alammar, Maarten Grootendorst: Ch. 12, fine-tuning generation models.