Chapter 5: Reinforcement learning and post-training

RLHF: PPO with a KL leash

RLHF (reinforcement learning from human feedback) trains a chat model toward high reward-model scores. With no limits, the model finds odd text that the score likes too much. You will compute, by hand, the two limits that hold it back, and explain them the way interviewers expect.

20 minInteractive lessonChapter 5: Reinforcement learning and post-trainingLesson 9 of 13

What you will learn

  • Write PPO's clipped surrogate objective
  • Add a per-token KL penalty to the reward
  • Explain reward hacking and the role of the reference model

Key terms

RLHF
reinforcement learning from human feedback: you train a chat model toward high scores
policy
the chat model that is in training; it picks the next token
advantage $A$
the reward from a token onward minus a guess of it
token
a word or part of a word
SFT model
the chat model after fine-tuning on good example replies, before reinforcement learning

Prerequisites

Take From demonstrations to preferences: reward models first. It comes right before this lesson in the course.

Where this lesson picks up: You built a reward model that turns human comparisons into one score for a full reply. Now the chat model must learn to raise that score. How do we train it safely?

What comes next

We needed a reward model and a whole training run to use human comparisons, so could we skip both?

Next lesson: DPO and its family

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.