Chapter 5: Reinforcement learning and post-training

GRPO: group-relative advantages and the KL estimator

You will compute GRPO's group advantages and its drift estimate by hand, and code its loss. GRPO trained DeepSeek-R1, and post-training interviews often ask how it differs from PPO.

16 minInteractive lessonChapter 5: Reinforcement learning and post-trainingLesson 12 of 13

What you will learn

  • Compute group-normalized advantages
  • Compute the k3 KL estimator
  • Assemble the clipped GRPO objective

Key terms

group
the G answers the model writes for one question
group average
the group's rewards added up and divided by G, used as the baseline
GRPO
Group Relative Policy Optimization: PPO with the group average in place of a critic
k3
a drift score for one sampled answer that is never below 0

Prerequisites

Take Policy gradient for LLMs: reward correct answers first. It comes right before this lesson in the course.

Where this lesson picks up: You trained a model with REINFORCE, which raises each sampled answer's log-chance in proportion to its advantage $r - b$. PPO got the baseline $b$ from a critic, a second trained network. Can you get $b$ more cheaply?

What comes next

Next: what happens to a group where every answer is wrong?

Next lesson: RL pitfalls: all-fail groups, entropy collapse and contaminated evals

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.