Chapter 5: Reinforcement learning and post-training
GRPO: group-relative advantages and the KL estimator
You will compute GRPO's group advantages and its drift estimate by hand, and code its loss. GRPO trained DeepSeek-R1, and post-training interviews often ask how it differs from PPO.
What you will learn
- Compute group-normalized advantages
- Compute the k3 KL estimator
- Assemble the clipped GRPO objective
Key terms
- group
- the G answers the model writes for one question
- group average
- the group's rewards added up and divided by G, used as the baseline
- GRPO
- Group Relative Policy Optimization: PPO with the group average in place of a critic
- k3
- a drift score for one sampled answer that is never below 0
Prerequisites
Take Policy gradient for LLMs: reward correct answers first. It comes right before this lesson in the course.
Where this lesson picks up: You trained a model with REINFORCE, which raises each sampled answer's log-chance in proportion to its advantage $r - b$. PPO got the baseline $b$ from a critic, a second trained network. Can you get $b$ more cheaply?
What comes next
Next: what happens to a group where every answer is wrong?
Next lesson: RL pitfalls: all-fail groups, entropy collapse and contaminated evals
More lessons in this part
- Policy gradient for LLMs: reward correct answers 24 min
- RL pitfalls: all-fail groups, entropy collapse and contaminated evals 18 min
Take the lesson
Short cards, one idea at a time, with quick checks as you go.