Chapter 5: Reinforcement learning and post-training

From demonstrations to preferences: reward models

You will write the pairwise loss of a reward model, train and check it, and explain what mixed labels do to it. The next step of post-training needs this score, and interviewers often ask for this loss.

19 minInteractive lessonChapter 5: Reinforcement learning and post-trainingLesson 8 of 13

What you will learn

  • Write the Bradley-Terry pairwise loss
  • Train and validate a reward model on preference pairs
  • Explain label noise and annotator disagreement in preference data

Key terms

SFT
supervised fine-tuning: training a model to copy example answers
reward model
a model that reads a prompt and one answer and returns one score
chosen answer
the answer a person picks from two answers
rejected answer
the answer a person does not pick
gap $\Delta$
the chosen score minus the rejected score

Prerequisites

Take LoRA and QLoRA: low-rank updates and memory math first. It comes right before this lesson in the course.

Where this lesson picks up: You fine-tuned a large model cheaply with LoRA and QLoRA. That training still copied example answers, so how do you train on a person's pick between two answers?

What comes next

The next lesson uses this score as a reward in PPO (Proximal Policy Optimization), a method that trains the answer-writing model to earn higher scores.

Next lesson: RLHF: PPO with a KL leash

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.