Chapter 5: Reinforcement learning and post-training
RL pitfalls: all-fail groups, entropy collapse and contaminated evals
An RL run can fool you in three ways: groups with no signal, a policy that spreads out and collapses, and a leaked test set. Interviewers who ask you to design or debug an RL run want to hear all three.
What you will learn
- Explain why all-fail and all-pass groups produce no learning signal
- Detect entropy collapse and name mitigations
- Check evaluation sets for contamination
Key terms
- all-fail group
- a group where every sampled answer fails, so every reward is 0
- entropy
- how spread out a policy's choices are, measured with the natural log
- n-gram
- a run of n words in a row
- contamination
- test questions that also appear in the training data
Prerequisites
Take GRPO: group-relative advantages and the KL estimator first. It comes right before this lesson in the course.
Where this lesson picks up: Last lesson GRPO gave each answer an advantage: its reward minus the group average, divided by the group's spread. That works when a group holds both passes and fails. What happens when the tasks are so hard that almost every answer fails?
What comes next
Next chapter: how do you make a model's reply follow an exact format, such as JSON?
Next lesson: Structured output: JSON schemas, constrained decoding and repair
More lessons in this part
- Policy gradient for LLMs: reward correct answers 24 min
- GRPO: group-relative advantages and the KL estimator 16 min
Take the lesson
Short cards, one idea at a time, with quick checks as you go.