Chapter 5: Reinforcement learning and post-training

Learning from experience: TD error, Q-learning and SARSA

A robot or a game player learns only from the steps it lives. This lesson turns the Bellman equation into a rule that learns from one step at a time. Interviewers ask you to compute one such update by hand and to explain the one difference between Q-learning and SARSA.

18 minInteractive lessonChapter 5: Reinforcement learning and post-trainingLesson 2 of 12

What you will learn

  • Compute a TD error and a TD(0) update by hand
  • Run a Q-learning update and a SARSA update on the same step and explain the difference
  • Say what on-policy and off-policy mean

Key terms

V(s)
the agent's guess of the total future reward from state s
discount γ
how much a reward one step later still counts
TD target
the reward plus γ times the next state's guess
TD error δ
the TD target minus the old guess

Prerequisites

Take Value iteration and the greedy policy first. It comes right before this lesson in the course.

Where this lesson picks up: You wrote a state's value from the next state's value with the Bellman equation, and value iteration solved it by repeating it. But that needs the chance of every next state, and a real agent almost never knows those chances.

What comes next

Q-learning takes a random action 10% of the time to explore, and the next lesson asks if that is the best way to explore.

Next lesson: Bandits: regret, epsilon-greedy and UCB

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.