PPO Clipped Surrogate Loss with Clip Fraction

MediumLLMs & GenAI

Problem statement

Implement a function that computes the Proximal Policy Optimization (PPO) clipped surrogate loss along with a diagnostic clip fraction metric.

In PPO, the policy is updated by optimizing a clipped surrogate objective that prevents the new policy from deviating too far from the old policy. The function should:

  1. Accept arrays of log-probabilities under the old and new policies, advantage estimates, and a clipping parameter epsilon
  2. Compute the probability ratios between the new and old policies
  3. Apply the PPO clipping mechanism to form the conservative surrogate objective
  4. Return a tuple of (loss, clip_fraction) where:
    • loss is the mean negated clipped surrogate objective (a scalar suitable for minimization)
    • clip_fraction is the proportion of samples where the probability ratio fell outside the range $[1 - \epsilon, 1 + \epsilon]$

Both values should be rounded to 4 decimal places.

The clip fraction is a commonly monitored training diagnostic: values near 0 suggest the policy barely changed, while values near 1 indicate the clipping constraint was heavily active.

Example

Input:

old_log_probs = np.array([-1.0, -2.0])
new_log_probs = np.array([-0.5, -2.5])
advantages = np.array([1.0, -1.0])
epsilon = 0.2

Output:

(-0.2, 1.0)

Reasoning: The ratios are $r = [e^{0.5}, e^{-0.5}] = [1.6487, 0.6065]$, both outside $[0.8, 1.2]$, so the clip fraction is $1.0$. Sample 1 (positive advantage): $\min(1.6487, 1.2) = 1.2$. Sample 2 (negative advantage): $\min(-0.6065, -0.8) = -0.8$. The mean objective is $(1.2 - 0.8)/2 = 0.2$, so the loss is $-0.2$.

Show hint

How do you turn two log-probabilities of the same action into a ratio of probabilities without leaving log space early?

Relevant roles: LLM Engineer, Research Scientist, ML Engineer, Applied Scientist and Gen AI Engineer.

Related free problems