Direct Preference Optimization (DPO) Loss

MediumML Coding

Problem statement

Implement dpo_loss(log_probs_chosen_policy, log_probs_rejected_policy, log_probs_chosen_ref, log_probs_rejected_ref, beta), the Direct Preference Optimization (DPO) loss used to align language models with human preferences.

DPO optimizes a policy directly on preference pairs (a chosen and a rejected response to the same prompt) without training a separate reward model. It uses the sequence log-probabilities of both responses under the current policy and under a frozen reference model.

Inputs (all lists of floats of the same length, the batch size):

  • log_probs_chosen_policy, log_probs_rejected_policy: log-probabilities of the chosen and rejected responses under the policy.
  • log_probs_chosen_ref, log_probs_rejected_ref: the same under the reference model.
  • beta: a positive temperature controlling the strength of the implicit KL constraint.

Return a dict with:

  • 'loss': the mean DPO loss over the batch, rounded to 4 decimals.
  • 'chosen_rewards' and 'rejected_rewards': lists of the implicit rewards of the chosen and rejected responses, each rounded to 4 decimals.

The implicit reward of a response is beta times the log-ratio of its policy probability to its reference probability. The DPO loss pushes the chosen response's implicit reward above the rejected one's: per pair it is $-\log \sigma(r_{\text{chosen}} - r_{\text{rejected}})$, where $\sigma$ is the logistic sigmoid.

Example

Input:

log_probs_chosen_policy=[-2.0, -1.5], log_probs_rejected_policy=[-3.0, -2.5], log_probs_chosen_ref=[-2.5, -2.0], log_probs_rejected_ref=[-2.5, -2.0], beta=0.5

Output:

{'loss': 0.4741, 'chosen_rewards': [0.25, 0.25], 'rejected_rewards': [-0.25, -0.25]}

Reasoning: For each sample, compute the log-ratio between policy and reference: chosen_log_ratios = [0.5, 0.5], rejected_log_ratios = [-0.5, -0.5]. The logits are beta (chosen_log_ratios - rejected_log_ratios) = 0.5 [1.0, 1.0] = [0.5, 0.5]. The loss per sample is -log(sigmoid(0.5)) = 0.4741. The implicit rewards are beta * log_ratios: chosen = [0.25, 0.25], rejected = [-0.25, -0.25]. Since the policy assigns relatively higher probability to chosen responses (compared to reference), the loss is below the uninformative baseline of 0.6931.

Show hint

For each response, what quantity compares how much more (or less) likely the policy makes it than the frozen reference does?

Relevant roles: LLM Engineer, Research Scientist, ML Engineer, Gen AI Engineer and Applied Scientist.

Related free problems