GRPO Objective Function

MediumML Coding

Problem statement

Implement the GRPO (Group Relative Policy Optimization) objective used to fine-tune language models with reinforcement learning. Compute it from likelihood ratios, advantages and per-sample policy probabilities.

The objective combines a PPO-style clipped surrogate with an importance-weighted KL penalty toward a reference policy. For each sample $i$ in the group:

  • clipped surrogate: $\min(\rho_i A_i, \ \text{clip}(\rho_i, 1 - \epsilon, 1 + \epsilon) A_i)$
  • KL estimate: $D_{KL}^{(i)} = \rho_i \left( \frac{\pi_{ref}(o_i)}{\pi_\theta(o_i)} - \log \frac{\pi_{ref}(o_i)}{\pi_\theta(o_i)} - 1 \right)$, where $\pi_\theta(o_i) = \rho_i \cdot \pi_{\theta_{old}}(o_i)$ is the current policy probability

The objective is the mean over samples of the clipped surrogate minus $\beta \cdot D_{KL}^{(i)}$.

Implement grpo_objective(rhos, A, pi_theta_old, pi_theta_ref, epsilon=0.2, beta=0.01) and return the objective as a float. The inputs pi_theta_old and pi_theta_ref are per-sample likelihood values, not distributions: do not normalize them. Clamp $\pi_\theta$ and $\pi_{ref}$ below at $10^{-10}$ before taking the ratio for numerical safety.

Example

Input:

grpo_objective([1.2, 0.8, 1.1], [1.0, 1.0, 1.0], [0.9, 1.1, 1.0], [1.0, 0.5, 1.5], epsilon=0.2, beta=0.01)

Output:

1.03277

Reasoning: The function computes the clipped surrogate objective and subtracts the importance-weighted KL penalty. The KL uses the estimator: rho * (r - log(r) - 1) where r = pi_ref / pi_theta.

Show hint

Which PPO mechanism keeps a single update from moving the policy too far, and why does it take a minimum?

Relevant roles: LLM Engineer, Research Scientist, ML Engineer, Gen AI Engineer and Applied Scientist.

Related free problems