LLM Lab

RLHF & DPO

A pretrained model can continue text, but not necessarily the way people want. Preference tuning teaches it which of two answers people prefer.

Where this sits in the life of an LLM
Pretraining at scale→Supervised fine-tuning→Preference tuning (RLHF, DPO)→Serving (KV cache, quantisation)

Preference data

For each prompt, show people two model answers and record which is better. That is the raw material: thousands of (prompt, chosen, rejected) triples. Try labelling a few in the lab.

promptExplain what a KV cache is in one sentence.
chosenIt stores the keys and values of earlier tokens so each new token does not recompute them.
rejectedA KV cache is a cache. It caches things to make stuff faster.
One preference pair: the same prompt with a better and a worse answer. RLHF trains a reward model on many such pairs; DPO trains the language model on them directly.

RLHF and DPO

RLHF first trains a reward model to predict the preferences, then uses reinforcement learning to make the language model produce high-reward answers, with a penalty for drifting too far from the original model.

DPO (direct preference optimisation) gets the same effect with one supervised-style loss: raise the probability of the chosen answer relative to the rejected one, measured against a frozen reference copy. The β setting controls how far the model may move.

A worked example

For the prompt “Explain what a KV cache is in one sentence”, a labeller prefers the precise answer over the vague one. RLHF first trains a reward model to score the preferred answer higher across many such pairs, then optimises the language model with reinforcement learning to earn high reward while staying close to its starting point. DPO skips the reward model: its loss directly raises the probability of the chosen answer relative to the rejected one, compared with a frozen reference model.

In practice

Preference tuning follows supervised fine-tuning. Llama 3, for example, used supervised fine-tuning, rejection sampling and DPO. Preference data can come from people or from another model acting as a judge, and its quality largely decides how helpful and how safe the final model is.

Easy to confuse

Check yourself · 1 of 3

What do RLHF and DPO learn from?

Check yourself · 2 of 3

What does DPO skip compared with classic RLHF?

Check yourself · 3 of 3

Why keep the tuned model close to the original (the KL term / β)?

Read more in the books

  • Reinforcement Learning from Human Feedback, Nathan Lambert: the chapters on reward modeling and direct alignment.
  • AI Engineering, Chip Huyen: Ch. 2, the section on post-training (supervised finetuning and preference finetuning).
  • Hands-On Large Language Models, Jay Alammar, Maarten Grootendorst: Ch. 12, fine-tuning generation models.
All books
Labpick the better answer
How to read this
  • Top: the two preference-tuning routes as stage maps. RLHF goes through a reward model and reinforcement learning; DPO trains directly on the pairs.
  • Pick the better answer in each pair; the count shows how many pairs you have labelled.
  • After labelling, the simulated training shows the model’s probability of the chosen answers rising and of the rejected ones falling.