DPO Implicit Reward Computation
Problem statement
In Direct Preference Optimization (DPO), the implicit reward for a response is the log-probability ratio between the trained policy and a frozen reference model, scaled by a temperature $\beta$:
$$ r(x, y) = \beta \cdot (\log \pi_\theta(y \mid x) - \log \pi_{ref}(y \mid x)) $$
Given the summed log-probabilities (over response tokens) of a chosen and a rejected response under both the policy and the reference model, compute the implicit rewards of both responses.
Implement dpo_rewards(policy_chosen_logps, policy_rejected_logps, ref_chosen_logps, ref_rejected_logps, beta). The four log-probability inputs are floats or numpy arrays of equal shape; beta is a float. Return a tuple (chosen_reward, rejected_reward):
- if the inputs are scalars, return Python floats;
- if the inputs are numpy arrays, return lists via
.tolist().
Do not round inside the function; the tests round to 4 decimals.
Example
Input:
policy_chosen_logps = -2.0, policy_rejected_logps = -3.0
ref_chosen_logps = -2.5, ref_rejected_logps = -2.5, beta = 0.1
Output:
(0.05, -0.05)
Reasoning: chosen reward $= 0.1 \cdot (-2.0 - (-2.5)) = 0.05$ and rejected reward $= 0.1 \cdot (-3.0 - (-2.5)) = -0.05$. The policy likes the chosen response more than the reference does and the rejected one less.
Show hint
In DPO, how is the reward expressed in terms of the policy and the reference model rather than a separate reward network?
Relevant roles: LLM Engineer, Research Scientist, Gen AI Engineer, ML Engineer and Applied Scientist.
Related free problems
- GRPO Objective FunctionMedium
- Multi-Head AttentionMedium
- Masked Self-AttentionMedium