PPO Clipped Surrogate Loss with Clip Fraction
Problem statement
Implement a function that computes the Proximal Policy Optimization (PPO) clipped surrogate loss along with a diagnostic clip fraction metric.
In PPO, the policy is updated by optimizing a clipped surrogate objective that prevents the new policy from deviating too far from the old policy. The function should:
- Accept arrays of log-probabilities under the old and new policies, advantage estimates, and a clipping parameter epsilon
- Compute the probability ratios between the new and old policies
- Apply the PPO clipping mechanism to form the conservative surrogate objective
- Return a tuple of (loss, clip_fraction) where:
lossis the mean negated clipped surrogate objective (a scalar suitable for minimization)clip_fractionis the proportion of samples where the probability ratio fell outside the range $[1 - \epsilon, 1 + \epsilon]$
Both values should be rounded to 4 decimal places.
The clip fraction is a commonly monitored training diagnostic: values near 0 suggest the policy barely changed, while values near 1 indicate the clipping constraint was heavily active.
Example
Input:
old_log_probs = np.array([-1.0, -2.0])
new_log_probs = np.array([-0.5, -2.5])
advantages = np.array([1.0, -1.0])
epsilon = 0.2
Output:
(-0.2, 1.0)
Reasoning: The ratios are $r = [e^{0.5}, e^{-0.5}] = [1.6487, 0.6065]$, both outside $[0.8, 1.2]$, so the clip fraction is $1.0$. Sample 1 (positive advantage): $\min(1.6487, 1.2) = 1.2$. Sample 2 (negative advantage): $\min(-0.6065, -0.8) = -0.8$. The mean objective is $(1.2 - 0.8)/2 = 0.2$, so the loss is $-0.2$.
Show hint
How do you turn two log-probabilities of the same action into a ratio of probabilities without leaving log space early?
Relevant roles: LLM Engineer, Research Scientist, ML Engineer, Applied Scientist and Gen AI Engineer.