Chapter 5: Reinforcement learning and post-training
RLHF: PPO with a KL leash
RLHF (reinforcement learning from human feedback) trains a chat model toward high reward-model scores. With no limits, the model finds odd text that the score likes too much. You will compute, by hand, the two limits that hold it back, and explain them the way interviewers expect.
What you will learn
- Write PPO's clipped surrogate objective
- Add a per-token KL penalty to the reward
- Explain reward hacking and the role of the reference model
Key terms
- RLHF
- reinforcement learning from human feedback: you train a chat model toward high scores
- policy
- the chat model that is in training; it picks the next token
- advantage $A$
- the reward from a token onward minus a guess of it
- token
- a word or part of a word
- SFT model
- the chat model after fine-tuning on good example replies, before reinforcement learning
Prerequisites
Take From demonstrations to preferences: reward models first. It comes right before this lesson in the course.
Where this lesson picks up: You built a reward model that turns human comparisons into one score for a full reply. Now the chat model must learn to raise that score. How do we train it safely?
What comes next
We needed a reward model and a whole training run to use human comparisons, so could we skip both?
Next lesson: DPO and its family
More lessons in this part
Take the lesson
Short cards, one idea at a time, with quick checks as you go.