Chapter 5: Reinforcement learning and post-training

Policy gradient for LLMs: reward correct answers

You will be able to derive the update that trains a model on checked answers, and run one step of it by hand. Maths and code tasks often have answers a program can check, but no worked solutions. Interviewers ask you to derive this update and to say how it differs from fine-tuning on written solutions.

24 minInteractive lessonChapter 5: Reinforcement learning and post-trainingLesson 11 of 13

What you will learn

  • Derive the REINFORCE gradient estimator
  • Explain baselines and advantages as variance reduction
  • Contrast RL with verifiable rewards against SFT on solutions

Key terms

policy $\pi$
the model's chances for each possible answer
logit $z$
a raw number for each answer, before softmax turns it into a chance
baseline $b$
a fixed number, not from the sampled answer, subtracted from the reward
advantage $A$
how much better an answer did than usual, $A = Q - V$

Prerequisites

Take DPO and its family first. It comes right before this lesson in the course.

Where this lesson picks up: You trained a model with DPO on pairs of preferred and rejected answers, with no reward model and no RL training stage. But many tasks have no answer pairs, only a program that checks if a final answer is right.

What comes next

Next, GRPO scores each answer against the other answers to the same question, with no extra model that predicts the reward.

Next lesson: GRPO: group-relative advantages and the KL estimator

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.