Chapter 5: Reinforcement learning and post-training

RL pitfalls: all-fail groups, entropy collapse and contaminated evals

An RL run can fool you in three ways: groups with no signal, a policy that spreads out and collapses, and a leaked test set. Interviewers who ask you to design or debug an RL run want to hear all three.

18 minInteractive lessonChapter 5: Reinforcement learning and post-trainingLesson 13 of 13

What you will learn

  • Explain why all-fail and all-pass groups produce no learning signal
  • Detect entropy collapse and name mitigations
  • Check evaluation sets for contamination

Key terms

all-fail group
a group where every sampled answer fails, so every reward is 0
entropy
how spread out a policy's choices are, measured with the natural log
n-gram
a run of n words in a row
contamination
test questions that also appear in the training data

Prerequisites

Take GRPO: group-relative advantages and the KL estimator first. It comes right before this lesson in the course.

Where this lesson picks up: Last lesson GRPO gave each answer an advantage: its reward minus the group average, divided by the group's spread. That works when a group holds both passes and fails. What happens when the tasks are so hard that almost every answer fails?

What comes next

Next chapter: how do you make a model's reply follow an exact format, such as JSON?

Next lesson: Structured output: JSON schemas, constrained decoding and repair

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.