RLOO: REINFORCE Leave-One-Out Advantages

MediumLLMs & GenAI

Problem statement

Implement REINFORCE Leave-One-Out (RLOO) advantages and the corresponding policy-gradient loss, a strong baseline for RL fine-tuning of LLMs that needs no learned critic.

Advantages. Given a group of $G$ sampled responses for the same prompt with rewards $r_1,\dots,r_G$:

$$b_i = \frac{1}{G-1}\sum_{j\neq i} r_j,\qquad A_i = r_i - b_i$$

If $G=1$, return [0.0]. Round each advantage to 6 decimals.

Loss (minimization form). With sequence log-probs $\ell_i=\log\pi_\theta(y_i|x)$:

$$\mathcal{L} = -\frac{1}{G}\sum_i A_i\,\ell_i$$

Optionally shape rewards with a KL penalty before computing advantages:

$$r_i' = r_i - \beta\cdot \mathrm{kl\_log\_ratios}_i$$

where kl_log_ratios[i] is approximately $\log\pi_\theta - \log\pi_{ref}$ for sample $i$.

Implement:

  • rloo_advantages(rewards) -> list[float]
  • rloo_loss(log_probs, rewards, kl_log_ratios=None, beta=0.0) -> float (rounded to 6 decimals)

Inputs may be Python lists or 1-D numpy arrays.

Example

Input:

rewards = [1.0, 2.0, 3.0]

Output:

[-1.5, 0.0, 1.5]

Reasoning: For $i=0$ the baseline is $(2+3)/2 = 2.5$, so $A_0 = -1.5$. For $i=1$ the baseline is $(1+3)/2 = 2$, so $A_1 = 0$. For $i=2$ the baseline is $(1+2)/2 = 1.5$, so $A_2 = 1.5$.

Show hint

Which baseline would be unbiased for sample $i$ yet still use the information in the rest of the group?

Relevant roles: LLM Engineer, Research Scientist, ML Engineer, Gen AI Engineer and Applied Scientist.

Related free problems