Chapter 5: Reinforcement learning and post-training

MDPs: discounted returns and the Bellman expectation equation

You will be able to compute a discounted return and check a table of state values with the Bellman expectation equation. Many reward-based methods, for game players and for chat assistants, build on this equation. Interviewers often ask you to write it from memory and to compute one step of it.

15 minInteractive lessonChapter 5: Reinforcement learning and post-trainingLesson 1 of 13

What you will learn

  • Compute a discounted return and explain what the discount factor does
  • Write the Bellman expectation equation for a state value and check it on a small chain

Key terms

agent
a player that picks one action at each step
reward
the points that one action earns
episode
one play of a game, from start to end
return G
the weighted total of the rewards in one episode

Prerequisites

Take Choosing a method: weight-only, W8A8, FP8, GPTQ, AWQ, SmoothQuant first. It comes right before this lesson in the course.

What comes next

Now you can check the values of a given policy. The next lesson asks how to find the best policy.

Next lesson: Value iteration and the greedy policy

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.