DPO Loss with NLL Regularization on Chosen Sequences

MediumML Coding

Problem statement

Implement a regularized variant of the Direct Preference Optimization (DPO) loss. The standard DPO loss compares the log-probability ratios of chosen and rejected responses under the policy model versus a frozen reference model, encouraging the policy to prefer the chosen response. To stabilize training and prevent the log-probability of chosen sequences from collapsing, an additional negative log-likelihood (NLL) regularization term is added on the chosen sequences with a scaling coefficient alpha.

Given the following inputs (all 1D numpy arrays of shape (batch_size,)):

  • policy_chosen_logps: sum of log-probabilities of chosen sequences under the policy model
  • policy_rejected_logps: sum of log-probabilities of rejected sequences under the policy model
  • ref_chosen_logps: sum of log-probabilities of chosen sequences under the reference model
  • ref_rejected_logps: sum of log-probabilities of rejected sequences under the reference model
  • chosen_avg_logps: per-token average log-probability of chosen sequences under the policy model (used to compute NLL)

And scalar hyperparameters beta (DPO temperature) and alpha (NLL coefficient).

Return the combined scalar loss as a Python float.

The DPO term per example is -log(sigmoid(beta * ((policy_chosen - ref_chosen) - (policy_rejected - ref_rejected)))). The total loss is the mean of the DPO term over the batch plus alpha times the mean NLL of chosen sequences (where NLL = -chosen_avg_logps).

Use a numerically stable implementation of -log(sigmoid(x)) (e.g., log(1 + exp(-x)) evaluated stably).

Example

Input:

policy_chosen_logps=[-2.0,-1.5], policy_rejected_logps=[-3.0,-2.5], ref_chosen_logps=[-2.5,-2.0], ref_rejected_logps=[-2.8,-2.3], chosen_avg_logps=[-0.5,-0.4], beta=0.1, alpha=0.2

Output:

0.7488

Reasoning: Chosen log-ratios = [0.5, 0.5]; rejected log-ratios = [-0.2, -0.2]; their difference = [0.7, 0.7]; logits = 0.1 [0.7, 0.7] = [0.07, 0.07]. The DPO term per example is log(1+exp(-0.07)) approximately 0.65875, mean approximately 0.65875. The NLL on chosen = -mean([-0.5, -0.4]) = 0.45. Total = 0.65875 + 0.20.45 approximately 0.7488.

Show hint

Which scalar per example compares how much the policy moved away from the reference on the chosen response versus the rejected one?

Relevant roles: LLM Engineer, Research Scientist, Gen AI Engineer, ML Engineer and Applied Scientist.

Related free problems