Reward Model Loss from Pairwise Preferences
Problem statement
In Reinforcement Learning from Human Feedback (RLHF), a reward model is trained to predict human preferences between pairs of responses. Given a dataset of preference pairs where humans have indicated which response they prefer (chosen) over another (rejected), the reward model learns to assign higher scalar scores to preferred responses.
Implement a function that computes the reward model training loss and accuracy given arrays of reward scores for chosen and rejected responses.
The function should:
- Accept lists of reward scores for chosen (preferred) responses and rejected (non-preferred) responses of equal length
- Accept an optional margin parameter (default 0.0) that encourages a minimum gap between chosen and rejected scores
- Return a dictionary with two keys:
'loss': the average negative log-sigmoid of the score differences (with margin subtracted), $-\frac{1}{N}\sum_i \log\sigma(r_w^{(i)} - r_l^{(i)} - m)$, rounded to 4 decimal places'accuracy': the fraction of pairs where the chosen reward strictly exceeds the rejected reward (the margin does not affect accuracy), rounded to 4 decimal places
Your implementation should be numerically stable for both very large and very small score differences.
Example
Input:
chosen_rewards = [1.0, 2.0, 3.0]
rejected_rewards = [0.5, 1.5, 2.0]
margin = 0.0
Output:
{'loss': 0.4205, 'accuracy': 1.0}
Reasoning: The differences are $[0.5, 0.5, 1.0]$. Their log-sigmoids are $-0.4741$, $-0.4741$ and $-0.3133$, so the loss is $(0.4741 + 0.4741 + 0.3133)/3 = 0.4205$. Every chosen score is higher, so accuracy is $1.0$.
Show hint
Which probabilistic model turns a difference of two scalar scores into the probability that one response is preferred?
Relevant roles: LLM Engineer, ML Engineer, Research Scientist, Applied Scientist and Gen AI Engineer.
Related free problems
- Direct Preference Optimization (DPO) LossML CodingMedium