RLOO: REINFORCE Leave-One-Out Advantages
Problem statement
Implement REINFORCE Leave-One-Out (RLOO) advantages and the corresponding policy-gradient loss, a strong baseline for RL fine-tuning of LLMs that needs no learned critic.
Advantages. Given a group of $G$ sampled responses for the same prompt with rewards $r_1,\dots,r_G$:
$$b_i = \frac{1}{G-1}\sum_{j\neq i} r_j,\qquad A_i = r_i - b_i$$
If $G=1$, return [0.0]. Round each advantage to 6 decimals.
Loss (minimization form). With sequence log-probs $\ell_i=\log\pi_\theta(y_i|x)$:
$$\mathcal{L} = -\frac{1}{G}\sum_i A_i\,\ell_i$$
Optionally shape rewards with a KL penalty before computing advantages:
$$r_i' = r_i - \beta\cdot \mathrm{kl\_log\_ratios}_i$$
where kl_log_ratios[i] is approximately $\log\pi_\theta - \log\pi_{ref}$ for sample $i$.
Implement:
rloo_advantages(rewards) -> list[float]rloo_loss(log_probs, rewards, kl_log_ratios=None, beta=0.0) -> float(rounded to 6 decimals)
Inputs may be Python lists or 1-D numpy arrays.
Example
Input:
rewards = [1.0, 2.0, 3.0]
Output:
[-1.5, 0.0, 1.5]
Reasoning: For $i=0$ the baseline is $(2+3)/2 = 2.5$, so $A_0 = -1.5$. For $i=1$ the baseline is $(1+3)/2 = 2$, so $A_1 = 0$. For $i=2$ the baseline is $(1+2)/2 = 1.5$, so $A_2 = 1.5$.
Show hint
Which baseline would be unbiased for sample $i$ yet still use the information in the rest of the group?
Relevant roles: LLM Engineer, Research Scientist, ML Engineer, Gen AI Engineer and Applied Scientist.