Decisions and ranking
Q-learning and DQN
Like learning which moves lead to winning a board game, Q-learning estimates how good each action is in a situation. Its action value includes both the next reward and discounted future rewards. DQN replaces a table of values with a neural network.
How it works
- Choose an action with exploration and observe the reward, next state, and whether the task terminated.
- Form the target as reward plus the discounted maximum next-state action value, omitting the future term at a true terminal state.
- Move the selected state-action estimate toward that target.
- For DQN, train on replayed transitions with a separate target network to reduce instability.
$$Q(s,a)\leftarrow Q(s,a)+\alpha\left[r+\gamma(1-d)\max_{a\prime}Q(s\prime,a\prime)-Q(s,a)\right]$$
The knobs that matter
- learning rate
- controls how far each update moves the value estimate
- discount factor
- sets the weight of future rewards
- exploration rate
- controls random action selection during data collection
- target update interval
- controls how often DQN refreshes its target network
What interviewers ask
- How does Q-learning differ from an on-policy action-value update?
- Why does DQN use a replay buffer and a target network?
- How would you handle terminal transitions in the target?
- When would you choose Q-learning over policy gradients?
Common mistakes
- Bootstrapping from a terminal state instead of using reward alone.
- Backpropagating through the DQN target rather than treating it as fixed for the update.
- Assuming a neural Q-function has the same convergence guarantees as tabular Q-learning.
When to use it
Use tabular Q-learning for small discrete problems and DQN for larger observations with manageable discrete actions. Continuous actions make the maximizing step harder and often favor actor-based methods.
Implement it (9 problems)
- Q-Learning for a Finite MDPProMediumSolve
- Fitted Value IterationProMediumSolve
- Asynchronous Value IterationProMediumSolve