DPO Implicit Reward Computation

EasyML Coding

Problem statement

In Direct Preference Optimization (DPO), the implicit reward for a response is the log-probability ratio between the trained policy and a frozen reference model, scaled by a temperature $\beta$:

$$ r(x, y) = \beta \cdot (\log \pi_\theta(y \mid x) - \log \pi_{ref}(y \mid x)) $$

Given the summed log-probabilities (over response tokens) of a chosen and a rejected response under both the policy and the reference model, compute the implicit rewards of both responses.

Implement dpo_rewards(policy_chosen_logps, policy_rejected_logps, ref_chosen_logps, ref_rejected_logps, beta). The four log-probability inputs are floats or numpy arrays of equal shape; beta is a float. Return a tuple (chosen_reward, rejected_reward):

  • if the inputs are scalars, return Python floats;
  • if the inputs are numpy arrays, return lists via .tolist().

Do not round inside the function; the tests round to 4 decimals.

Example

Input:

policy_chosen_logps = -2.0, policy_rejected_logps = -3.0
ref_chosen_logps = -2.5, ref_rejected_logps = -2.5, beta = 0.1

Output:

(0.05, -0.05)

Reasoning: chosen reward $= 0.1 \cdot (-2.0 - (-2.5)) = 0.05$ and rejected reward $= 0.1 \cdot (-3.0 - (-2.5)) = -0.05$. The policy likes the chosen response more than the reference does and the rejected one less.

Show hint

In DPO, how is the reward expressed in terms of the policy and the reference model rather than a separate reward network?

Relevant roles: LLM Engineer, Research Scientist, Gen AI Engineer, ML Engineer and Applied Scientist.

Related free problems