Log-Probability Ratio for Token Sequences
Problem statement
In reinforcement learning from human feedback (RLHF) and related alignment methods, a critical quantity is the log-probability ratio between a policy model and a reference model evaluated over a sequence of tokens. This ratio measures how much the policy has diverged from the reference on each generated token.
Given token-level log-probabilities from a policy model and a reference model for a batch of sequences (which may have different lengths due to padding), implement a function that computes:
- Per-token log-probability ratios: For each token position, compute the difference between the policy log-prob and the reference log-prob. Padded positions (where mask is 0) should have a ratio of 0.
- Sequence-level log-probability ratios: For each sequence in the batch, sum the per-token log-ratios over all valid (non-padded) tokens.
- Mean sequence log-probability ratio: The average of the sequence-level ratios across the batch.
Your function should accept:
policy_log_probs: a numpy array of shape(B, T)containing log-probabilities under the policyreference_log_probs: a numpy array of shape(B, T)containing log-probabilities under the reference modelmask: a numpy array of shape(B, T)with 1 for real tokens and 0 for padding
Return a dictionary with keys 'per_token_log_ratios' (shape (B, T)), 'sequence_log_ratios' (shape (B,)), and 'mean_sequence_log_ratio' (a float scalar).
Example
Input:
policy_log_probs = np.array([[-1.0, -2.0, -0.5], [-1.5, -0.8, -1.2]])
reference_log_probs = np.array([[-1.2, -1.8, -0.7], [-1.0, -1.0, -1.5]])
mask = np.array([[1, 1, 1], [1, 1, 1]])
Output:
{'per_token_log_ratios': [[0.2, -0.2, 0.2], [-0.5, 0.2, 0.3]],
'sequence_log_ratios': [0.2, 0.0],
'mean_sequence_log_ratio': 0.1}
(arrays shown as lists)
Reasoning: Sequence 0 has per-token ratios $-1.0 - (-1.2) = 0.2$, $-2.0 - (-1.8) = -0.2$ and $-0.5 - (-0.7) = 0.2$, which sum to $0.2$. Sequence 1 has $-0.5$, $0.2$ and $0.3$, which sum to $0.0$. The batch mean is $(0.2 + 0.0)/2 = 0.1$.
Show hint
Since a sequence's probability is a product of token probabilities, what does a ratio of two sequence probabilities become in log space?
Relevant roles: LLM Engineer, Gen AI Engineer, ML Engineer and Research Scientist.
Related free problems
- DPO Loss with NLL Regularization on Chosen SequencesML CodingMedium