Log-Probability Ratio for Token Sequences

EasyLLMs & GenAI

Problem statement

In reinforcement learning from human feedback (RLHF) and related alignment methods, a critical quantity is the log-probability ratio between a policy model and a reference model evaluated over a sequence of tokens. This ratio measures how much the policy has diverged from the reference on each generated token.

Given token-level log-probabilities from a policy model and a reference model for a batch of sequences (which may have different lengths due to padding), implement a function that computes:

  1. Per-token log-probability ratios: For each token position, compute the difference between the policy log-prob and the reference log-prob. Padded positions (where mask is 0) should have a ratio of 0.
  1. Sequence-level log-probability ratios: For each sequence in the batch, sum the per-token log-ratios over all valid (non-padded) tokens.
  1. Mean sequence log-probability ratio: The average of the sequence-level ratios across the batch.

Your function should accept:

  • policy_log_probs: a numpy array of shape (B, T) containing log-probabilities under the policy
  • reference_log_probs: a numpy array of shape (B, T) containing log-probabilities under the reference model
  • mask: a numpy array of shape (B, T) with 1 for real tokens and 0 for padding

Return a dictionary with keys 'per_token_log_ratios' (shape (B, T)), 'sequence_log_ratios' (shape (B,)), and 'mean_sequence_log_ratio' (a float scalar).

Example

Input:

policy_log_probs = np.array([[-1.0, -2.0, -0.5], [-1.5, -0.8, -1.2]])
reference_log_probs = np.array([[-1.2, -1.8, -0.7], [-1.0, -1.0, -1.5]])
mask = np.array([[1, 1, 1], [1, 1, 1]])

Output:

{'per_token_log_ratios': [[0.2, -0.2, 0.2], [-0.5, 0.2, 0.3]],
 'sequence_log_ratios': [0.2, 0.0],
 'mean_sequence_log_ratio': 0.1}
(arrays shown as lists)

Reasoning: Sequence 0 has per-token ratios $-1.0 - (-1.2) = 0.2$, $-2.0 - (-1.8) = -0.2$ and $-0.5 - (-0.7) = 0.2$, which sum to $0.2$. Sequence 1 has $-0.5$, $0.2$ and $0.3$, which sum to $0.0$. The batch mean is $(0.2 + 0.0)/2 = 0.1$.

Show hint

Since a sequence's probability is a product of token probabilities, what does a ratio of two sequence probabilities become in log space?

Relevant roles: LLM Engineer, Gen AI Engineer, ML Engineer and Research Scientist.

Related free problems