Chapter 5: Reinforcement learning and post-training
Policy gradient for LLMs: reward correct answers
You will be able to derive the update that trains a model on checked answers, and run one step of it by hand. Maths and code tasks often have answers a program can check, but no worked solutions. Interviewers ask you to derive this update and to say how it differs from fine-tuning on written solutions.
What you will learn
- Derive the REINFORCE gradient estimator
- Explain baselines and advantages as variance reduction
- Contrast RL with verifiable rewards against SFT on solutions
Key terms
- policy $\pi$
- the model's chances for each possible answer
- logit $z$
- a raw number for each answer, before softmax turns it into a chance
- baseline $b$
- a fixed number, not from the sampled answer, subtracted from the reward
- advantage $A$
- how much better an answer did than usual, $A = Q - V$
Prerequisites
Take DPO and its family first. It comes right before this lesson in the course.
Where this lesson picks up: You trained a model with DPO on pairs of preferred and rejected answers, with no reward model and no RL training stage. But many tasks have no answer pairs, only a program that checks if a final answer is right.
What comes next
Next, GRPO scores each answer against the other answers to the same question, with no extra model that predicts the reward.
Next lesson: GRPO: group-relative advantages and the KL estimator
More lessons in this part
- GRPO: group-relative advantages and the KL estimator 16 min
- RL pitfalls: all-fail groups, entropy collapse and contaminated evals 18 min
Take the lesson
Short cards, one idea at a time, with quick checks as you go.