Chapter 5: Reinforcement learning and post-training
From demonstrations to preferences: reward models
You will write the pairwise loss of a reward model, train and check it, and explain what mixed labels do to it. The next step of post-training needs this score, and interviewers often ask for this loss.
What you will learn
- Write the Bradley-Terry pairwise loss
- Train and validate a reward model on preference pairs
- Explain label noise and annotator disagreement in preference data
Key terms
- SFT
- supervised fine-tuning: training a model to copy example answers
- reward model
- a model that reads a prompt and one answer and returns one score
- chosen answer
- the answer a person picks from two answers
- rejected answer
- the answer a person does not pick
- gap $\Delta$
- the chosen score minus the rejected score
Prerequisites
Take LoRA and QLoRA: low-rank updates and memory math first. It comes right before this lesson in the course.
Where this lesson picks up: You fine-tuned a large model cheaply with LoRA and QLoRA. That training still copied example answers, so how do you train on a person's pick between two answers?
What comes next
The next lesson uses this score as a reward in PPO (Proximal Policy Optimization), a method that trains the answer-writing model to earn higher scores.
Next lesson: RLHF: PPO with a KL leash
More lessons in this part
- RLHF: PPO with a KL leash 20 min
- DPO and its family 16 min
Take the lesson
Short cards, one idea at a time, with quick checks as you go.