GRPO Objective Function
Problem statement
Implement the GRPO (Group Relative Policy Optimization) objective used to fine-tune language models with reinforcement learning. Compute it from likelihood ratios, advantages and per-sample policy probabilities.
The objective combines a PPO-style clipped surrogate with an importance-weighted KL penalty toward a reference policy. For each sample $i$ in the group:
- clipped surrogate: $\min(\rho_i A_i, \ \text{clip}(\rho_i, 1 - \epsilon, 1 + \epsilon) A_i)$
- KL estimate: $D_{KL}^{(i)} = \rho_i \left( \frac{\pi_{ref}(o_i)}{\pi_\theta(o_i)} - \log \frac{\pi_{ref}(o_i)}{\pi_\theta(o_i)} - 1 \right)$, where $\pi_\theta(o_i) = \rho_i \cdot \pi_{\theta_{old}}(o_i)$ is the current policy probability
The objective is the mean over samples of the clipped surrogate minus $\beta \cdot D_{KL}^{(i)}$.
Implement grpo_objective(rhos, A, pi_theta_old, pi_theta_ref, epsilon=0.2, beta=0.01) and return the objective as a float. The inputs pi_theta_old and pi_theta_ref are per-sample likelihood values, not distributions: do not normalize them. Clamp $\pi_\theta$ and $\pi_{ref}$ below at $10^{-10}$ before taking the ratio for numerical safety.
Example
Input:
grpo_objective([1.2, 0.8, 1.1], [1.0, 1.0, 1.0], [0.9, 1.1, 1.0], [1.0, 0.5, 1.5], epsilon=0.2, beta=0.01)
Output:
1.03277
Reasoning: The function computes the clipped surrogate objective and subtracts the importance-weighted KL penalty. The KL uses the estimator: rho * (r - log(r) - 1) where r = pi_ref / pi_theta.
Show hint
Which PPO mechanism keeps a single update from moving the policy too far, and why does it take a minimum?
Relevant roles: LLM Engineer, Research Scientist, ML Engineer, Gen AI Engineer and Applied Scientist.
Related free problems
- Multi-Head AttentionMedium
- Masked Self-AttentionMedium