Identity Preference Optimization (IPO) Loss
Problem statement
Implement the Identity Preference Optimization (IPO) loss (Azar et al., 2023).
IPO replaces DPO's logistic loss with a squared loss that targets a fixed log-ratio gap. Given sequence log-probabilities under the policy and a frozen reference model, define the preference margin
$$h_\theta(y_w, y_l) = \Big(\log \pi_\theta(y_w) - \log \pi_{\rm ref}(y_w)\Big) - \Big(\log \pi_\theta(y_l) - \log \pi_{\rm ref}(y_l)\Big)$$
and the IPO loss
$$\mathcal{L}_{\rm IPO} = \mathbb{E}\big[(h_\theta(y_w, y_l) - \tfrac{1}{2\beta})^2\big]$$
where $\beta > 0$ controls how strongly the policy may move away from the reference.
Implement ipo_loss(policy_chosen_logps, policy_rejected_logps, ref_chosen_logps, ref_rejected_logps, beta) returning the mean batch loss as a float rounded to 6 decimals. Inputs are equal-length lists or arrays.
Example
Input:
ipo_loss([-2.0, -1.5], [-3.0, -2.5], [-2.5, -2.0], [-2.8, -2.3], 0.1)
Output:
18.49
Reasoning: the chosen log-ratios are [0.5, 0.5] and the rejected log-ratios are [-0.2, -0.2], so $h = 0.7$ for both rows. The target is $1/(2 \cdot 0.1) = 5$, so the loss is the mean of $(0.7 - 5)^2 = 18.49$.
Show hint
What quantity measures how much more the policy favors the chosen response than the reference model does?
Relevant roles: LLM Engineer, Gen AI Engineer, Research Scientist, ML Engineer and AI Engineer.
Related free problems
- Multi-Head AttentionMedium
- Masked Self-AttentionMedium