Decisions and ranking

Q-learning and DQN

Like learning which moves lead to winning a board game, Q-learning estimates how good each action is in a situation. Its action value includes both the next reward and discounted future rewards. DQN replaces a table of values with a neural network.

Current Q2Target1 + 0.9 × 4 = 4.6Updated Q2 + 0.5 × 2.6 = 3.3
One nonterminal update with reward 1, discount 0.9, next-state maximum 4, and learning rate 0.5.
Current Q2Target1 + 0.9 × 4 = 4.6Updated Q2 + 0.5 × 2.6 = 3.3
One nonterminal update with reward 1, discount 0.9, next-state maximum 4, and learning rate 0.5.

How it works

  1. Choose an action with exploration and observe the reward, next state, and whether the task terminated.
  2. Form the target as reward plus the discounted maximum next-state action value, omitting the future term at a true terminal state.
  3. Move the selected state-action estimate toward that target.
  4. For DQN, train on replayed transitions with a separate target network to reduce instability.

$$Q(s,a)\leftarrow Q(s,a)+\alpha\left[r+\gamma(1-d)\max_{a\prime}Q(s\prime,a\prime)-Q(s,a)\right]$$

The knobs that matter

learning rate
controls how far each update moves the value estimate
discount factor
sets the weight of future rewards
exploration rate
controls random action selection during data collection
target update interval
controls how often DQN refreshes its target network

What interviewers ask

  • How does Q-learning differ from an on-policy action-value update?
  • Why does DQN use a replay buffer and a target network?
  • How would you handle terminal transitions in the target?
  • When would you choose Q-learning over policy gradients?

Common mistakes

  • Bootstrapping from a terminal state instead of using reward alone.
  • Backpropagating through the DQN target rather than treating it as fixed for the update.
  • Assuming a neural Q-function has the same convergence guarantees as tabular Q-learning.

When to use it

Use tabular Q-learning for small discrete problems and DQN for larger observations with manageable discrete actions. Continuous actions make the maximizing step harder and often favor actor-based methods.

Implement it (9 problems)

Learn it

Interview questions about q-learning and dqn