Reward Model Loss from Pairwise Preferences

MediumLLMs & GenAI

Problem statement

In Reinforcement Learning from Human Feedback (RLHF), a reward model is trained to predict human preferences between pairs of responses. Given a dataset of preference pairs where humans have indicated which response they prefer (chosen) over another (rejected), the reward model learns to assign higher scalar scores to preferred responses.

Implement a function that computes the reward model training loss and accuracy given arrays of reward scores for chosen and rejected responses.

The function should:

  • Accept lists of reward scores for chosen (preferred) responses and rejected (non-preferred) responses of equal length
  • Accept an optional margin parameter (default 0.0) that encourages a minimum gap between chosen and rejected scores
  • Return a dictionary with two keys:
    • 'loss': the average negative log-sigmoid of the score differences (with margin subtracted), $-\frac{1}{N}\sum_i \log\sigma(r_w^{(i)} - r_l^{(i)} - m)$, rounded to 4 decimal places
    • 'accuracy': the fraction of pairs where the chosen reward strictly exceeds the rejected reward (the margin does not affect accuracy), rounded to 4 decimal places

Your implementation should be numerically stable for both very large and very small score differences.

Example

Input:

chosen_rewards = [1.0, 2.0, 3.0]
rejected_rewards = [0.5, 1.5, 2.0]
margin = 0.0

Output:

{'loss': 0.4205, 'accuracy': 1.0}

Reasoning: The differences are $[0.5, 0.5, 1.0]$. Their log-sigmoids are $-0.4741$, $-0.4741$ and $-0.3133$, so the loss is $(0.4741 + 0.4741 + 0.3133)/3 = 0.4205$. Every chosen score is higher, so accuracy is $1.0$.

Show hint

Which probabilistic model turns a difference of two scalar scores into the probability that one response is preferred?

Relevant roles: LLM Engineer, ML Engineer, Research Scientist, Applied Scientist and Gen AI Engineer.

Related free problems