Chapter 5: Reinforcement learning and post-training
MDPs: discounted returns and the Bellman expectation equation
You will be able to compute a discounted return and check a table of state values with the Bellman expectation equation. Many reward-based methods, for game players and for chat assistants, build on this equation. Interviewers often ask you to write it from memory and to compute one step of it.
What you will learn
- Compute a discounted return and explain what the discount factor does
- Write the Bellman expectation equation for a state value and check it on a small chain
Key terms
- agent
- a player that picks one action at each step
- reward
- the points that one action earns
- episode
- one play of a game, from start to end
- return G
- the weighted total of the rewards in one episode
Prerequisites
Take Choosing a method: weight-only, W8A8, FP8, GPTQ, AWQ, SmoothQuant first. It comes right before this lesson in the course.
What comes next
Now you can check the values of a given policy. The next lesson asks how to find the best policy.
Next lesson: Value iteration and the greedy policy
More lessons in this part
- Value iteration and the greedy policy 18 min
- Learning from experience: TD error, Q-learning and SARSA 18 min
- Bandits: regret, epsilon-greedy and UCB 16 min
- Thompson sampling: picking by sampling your beliefs 14 min
Take the lesson
Short cards, one idea at a time, with quick checks as you go.