Decisions and ranking

Policy gradient

Like practicing a move more often after it works better than expected, a policy gradient adjusts how likely actions are. It directly trains a policy using sampled outcomes. A baseline helps judge whether an outcome was better than expected.

Sample action Aprobability = 0.5Estimate returnreturn = 3Subtractbaseline3 − 2 = +1Update policyraise log P(A)
A sampled action with advantage +1 receives an update that encourages it in that state.
Sample action Aprobability = 0.5Estimate returnreturn = 3Subtract baseline3 − 2 = +1Update policyraise log P(A)
A sampled action with advantage +1 receives an update that encourages it in that state.

How it works

  1. Sample actions from the current policy and collect rewards along trajectories.
  2. Estimate discounted returns or advantages, optionally using a learned state-value baseline; here the objective is expected discounted episodic return.
  3. Weight each sampled action’s log-probability gradient by its advantage, encouraging positive advantages and discouraging negative ones.
  4. For actor-critic methods, also train the value estimator; for PPO, use a clipped probability-ratio objective to limit the update incentive.

$$\widehat{\nabla_\theta J}=\sum_{t=0}^{T-1}\gamma^t\nabla_\theta\log\pi_\theta(a_t\mid s_t)(G_t-b(s_t)),\qquad G_t=\sum_{k=t}^{T-1}\gamma^{k-t}r_k$$

The knobs that matter

learning rate
sets policy update size
baseline or advantage estimator
trades estimation bias against variance
entropy coefficient
encourages broader action distributions
PPO clip range
limits the surrogate incentive for large probability-ratio changes

What interviewers ask

  • Why does a state-dependent baseline leave the expected policy gradient unchanged, and how can it reduce variance?
  • How do actor-critic methods differ from basic episodic policy gradients?
  • What does PPO clip, and does clipping guarantee a small policy change?
  • How would you represent a policy over continuous actions?

Common mistakes

  • Reversing the loss sign and making rewarded actions less likely.
  • Letting the policy loss backpropagate through an advantage estimate that should be held fixed.
  • Treating PPO clipping as a hard bound on the policy’s actual change.

When to use it

Use policy gradients for stochastic policies and continuous action spaces where maximizing a Q-function is awkward. They can require many samples and careful variance control when rewards are sparse or noisy.

Implement it (11 problems)

Learn it

Interview questions about policy gradient