Chapter 5: Reinforcement learning and post-training
DPO and its family
You will be able to derive the DPO loss on a whiteboard and compute it by hand from four log-probabilities. Teams with fixed preference data and little compute use DPO because it removes the reward model and the RL training. Interviewers often ask how DPO does this.
What you will learn
- Derive the DPO loss from the KL-constrained objective
- Compute the DPO loss from policy and reference log-probs
- Explain length bias and over-optimization and how variants (IPO, SimPO) respond
Key terms
- reference model $\pi_{\text{ref}}$
- the frozen SFT model that the new policy stays near
- KL weight $\beta$
- the weight of the KL penalty in the goal
- partition term $Z(x)$
- the total that makes the tilted chances add to 1, one per prompt
- log loss
- the loss for a yes-or-no choice, small when the model is confident and right
Prerequisites
Take RLHF: PPO with a KL leash first. It comes right before this lesson in the course.
Where this lesson picks up: You trained a policy with PPO to chase a reward model's score, and a KL penalty kept it near the SFT model. That needed a reward model, sampling and RL training, so can the preference pairs alone give the same result?
What comes next
If an answer is simply right or wrong, can you skip preference pairs and push up the log-probability of good sampled answers directly?
Next lesson: Policy gradient for LLMs: reward correct answers
More lessons in this part
Take the lesson
Short cards, one idea at a time, with quick checks as you go.