Decisions and ranking
Policy gradient
Like practicing a move more often after it works better than expected, a policy gradient adjusts how likely actions are. It directly trains a policy using sampled outcomes. A baseline helps judge whether an outcome was better than expected.
How it works
- Sample actions from the current policy and collect rewards along trajectories.
- Estimate discounted returns or advantages, optionally using a learned state-value baseline; here the objective is expected discounted episodic return.
- Weight each sampled action’s log-probability gradient by its advantage, encouraging positive advantages and discouraging negative ones.
- For actor-critic methods, also train the value estimator; for PPO, use a clipped probability-ratio objective to limit the update incentive.
$$\widehat{\nabla_\theta J}=\sum_{t=0}^{T-1}\gamma^t\nabla_\theta\log\pi_\theta(a_t\mid s_t)(G_t-b(s_t)),\qquad G_t=\sum_{k=t}^{T-1}\gamma^{k-t}r_k$$
The knobs that matter
- learning rate
- sets policy update size
- baseline or advantage estimator
- trades estimation bias against variance
- entropy coefficient
- encourages broader action distributions
- PPO clip range
- limits the surrogate incentive for large probability-ratio changes
What interviewers ask
- Why does a state-dependent baseline leave the expected policy gradient unchanged, and how can it reduce variance?
- How do actor-critic methods differ from basic episodic policy gradients?
- What does PPO clip, and does clipping guarantee a small policy change?
- How would you represent a policy over continuous actions?
Common mistakes
- Reversing the loss sign and making rewarded actions less likely.
- Letting the policy loss backpropagate through an advantage estimate that should be held fixed.
- Treating PPO clipping as a hard bound on the policy’s actual change.
When to use it
Use policy gradients for stochastic policies and continuous action spaces where maximizing a Q-function is awkward. They can require many samples and careful variance control when rewards are sparse or noisy.
Implement it (11 problems)
- One-step Advantage Actor-CriticProMediumSolve