Chapter 5: Reinforcement learning and post-training
Learning from experience: TD error, Q-learning and SARSA
A robot or a game player learns only from the steps it lives. This lesson turns the Bellman equation into a rule that learns from one step at a time. Interviewers ask you to compute one such update by hand and to explain the one difference between Q-learning and SARSA.
What you will learn
- Compute a TD error and a TD(0) update by hand
- Run a Q-learning update and a SARSA update on the same step and explain the difference
- Say what on-policy and off-policy mean
Key terms
- V(s)
- the agent's guess of the total future reward from state s
- discount γ
- how much a reward one step later still counts
- TD target
- the reward plus γ times the next state's guess
- TD error δ
- the TD target minus the old guess
Prerequisites
Take Value iteration and the greedy policy first. It comes right before this lesson in the course.
Where this lesson picks up: You wrote a state's value from the next state's value with the Bellman equation, and value iteration solved it by repeating it. But that needs the chance of every next state, and a real agent almost never knows those chances.
What comes next
Q-learning takes a random action 10% of the time to explore, and the next lesson asks if that is the best way to explore.
Next lesson: Bandits: regret, epsilon-greedy and UCB
More lessons in this part
- MDPs: discounted returns and the Bellman expectation equation 15 min
- Value iteration and the greedy policy 18 min
- Bandits: regret, epsilon-greedy and UCB 16 min
- Thompson sampling: picking by sampling your beliefs 14 min
Take the lesson
Short cards, one idea at a time, with quick checks as you go.