Chapter 5: Reinforcement learning and post-training

DPO and its family

You will be able to derive the DPO loss on a whiteboard and compute it by hand from four log-probabilities. Teams with fixed preference data and little compute use DPO because it removes the reward model and the RL training. Interviewers often ask how DPO does this.

16 minInteractive lessonChapter 5: Reinforcement learning and post-trainingLesson 10 of 13

What you will learn

  • Derive the DPO loss from the KL-constrained objective
  • Compute the DPO loss from policy and reference log-probs
  • Explain length bias and over-optimization and how variants (IPO, SimPO) respond

Key terms

reference model $\pi_{\text{ref}}$
the frozen SFT model that the new policy stays near
KL weight $\beta$
the weight of the KL penalty in the goal
partition term $Z(x)$
the total that makes the tilted chances add to 1, one per prompt
log loss
the loss for a yes-or-no choice, small when the model is confident and right

Prerequisites

Take RLHF: PPO with a KL leash first. It comes right before this lesson in the course.

Where this lesson picks up: You trained a policy with PPO to chase a reward model's score, and a KL penalty kept it near the SFT model. That needed a reward model, sampling and RL training, so can the preference pairs alone give the same result?

What comes next

If an answer is simply right or wrong, can you skip preference pairs and push up the log-probability of good sampled answers directly?

Next lesson: Policy gradient for LLMs: reward correct answers

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.