DPO Loss with NLL Regularization on Chosen Sequences
Problem statement
Implement a regularized variant of the Direct Preference Optimization (DPO) loss. The standard DPO loss compares the log-probability ratios of chosen and rejected responses under the policy model versus a frozen reference model, encouraging the policy to prefer the chosen response. To stabilize training and prevent the log-probability of chosen sequences from collapsing, an additional negative log-likelihood (NLL) regularization term is added on the chosen sequences with a scaling coefficient alpha.
Given the following inputs (all 1D numpy arrays of shape (batch_size,)):
policy_chosen_logps: sum of log-probabilities of chosen sequences under the policy modelpolicy_rejected_logps: sum of log-probabilities of rejected sequences under the policy modelref_chosen_logps: sum of log-probabilities of chosen sequences under the reference modelref_rejected_logps: sum of log-probabilities of rejected sequences under the reference modelchosen_avg_logps: per-token average log-probability of chosen sequences under the policy model (used to compute NLL)
And scalar hyperparameters beta (DPO temperature) and alpha (NLL coefficient).
Return the combined scalar loss as a Python float.
The DPO term per example is -log(sigmoid(beta * ((policy_chosen - ref_chosen) - (policy_rejected - ref_rejected)))). The total loss is the mean of the DPO term over the batch plus alpha times the mean NLL of chosen sequences (where NLL = -chosen_avg_logps).
Use a numerically stable implementation of -log(sigmoid(x)) (e.g., log(1 + exp(-x)) evaluated stably).
Example
Input:
policy_chosen_logps=[-2.0,-1.5], policy_rejected_logps=[-3.0,-2.5], ref_chosen_logps=[-2.5,-2.0], ref_rejected_logps=[-2.8,-2.3], chosen_avg_logps=[-0.5,-0.4], beta=0.1, alpha=0.2
Output:
0.7488
Reasoning: Chosen log-ratios = [0.5, 0.5]; rejected log-ratios = [-0.2, -0.2]; their difference = [0.7, 0.7]; logits = 0.1 [0.7, 0.7] = [0.07, 0.07]. The DPO term per example is log(1+exp(-0.07)) approximately 0.65875, mean approximately 0.65875. The NLL on chosen = -mean([-0.5, -0.4]) = 0.45. Total = 0.65875 + 0.20.45 approximately 0.7488.
Show hint
Which scalar per example compares how much the policy moved away from the reference on the chosen response versus the rejected one?
Relevant roles: LLM Engineer, Research Scientist, Gen AI Engineer, ML Engineer and Applied Scientist.
Related free problems
- Multi-Head AttentionMedium
- Masked Self-AttentionMedium