Debug a Left-Padded Causal LM Loss

HardLLMs & GenAI

Reported in interviews at OpenAI

Problem statement

You inherit the training code of a small decoder-only language model. Training is unreliable: every so often a step crashes with a shape error, most steps report a NaN loss, and on the few batches that do produce a number the loss looks plausible but the model learns badly. To isolate the problem, a teammate reduced the model to a single layer with one attention head and wrote the numpy loss below. It reproduces all three symptoms.

Intended computation. Batches are left-padded: each row of tokens ($B \times T$ integer ids) is some number of pad_id entries followed by the real tokens (a row may be all padding). E is the $V \times D$ token embedding, P the $T_{max} \times D$ position embedding ($T_{max} \ge T$), Wq, Wk, Wv are $D \times D$ and Wo is $D \times V$.

  • Positions count real tokens only: the first real token of every row has position 0, the next position 1, and so on (pad slots use position 0; their value never matters).
  • $x_t = E[\text{tok}_t] + P[\text{pos}_t]$; $q_t, k_t, v_t = x_t W_q, x_t W_k, x_t W_v$.
  • Query $t$ may attend to key $j$ only if $j \le t$ and token $j$ is real; the weights are a softmax of $q_t \cdot k_j / \sqrt{D}$ over the allowed keys. A query with no allowed key gets zero attention output.
  • $h_t = x_t + \sum_j a_{tj} v_j$ and $\text{logits}_t = h_t W_o$.
  • Position $t$ predicts token $t+1$. A target counts only if both token $t$ and token $t+1$ are real. The loss is the mean of $-\log \mathrm{softmax}(\text{logits}_t)[\text{tok}_{t+1}]$ over the counted targets.
  • Return [mean_loss, n_targets], or [0.0, 0] if the batch has no counted target.

The buggy code:

import numpy as np


def lm_loss(tokens, pad_id, E, P, Wq, Wk, Wv, Wo):
    tok = np.array(tokens).squeeze()
    E, P, Wq, Wk, Wv, Wo = (np.asarray(a, dtype=float) for a in (E, P, Wq, Wk, Wv, Wo))
    B, T = tok.shape
    D = E.shape[1]
    real = tok != pad_id
    pos = np.broadcast_to(np.arange(T), (B, T))
    X = E[tok] + P[pos]
    Q, K, V = X @ Wq, X @ Wk, X @ Wv
    scores = Q @ K.transpose(0, 2, 1) / np.sqrt(D)
    causal = np.tril(np.ones((T, T), dtype=bool))
    allowed = causal[None, :, :] & real[:, None, :]
    masked = np.where(allowed, scores, -np.inf)
    e = np.exp(masked - masked.max(axis=-1, keepdims=True))
    w = e / e.sum(axis=-1, keepdims=True)
    H = X + w @ V
    logits = H @ Wo
    z = logits[:, :-1, :]
    logp = z - np.log(np.exp(z).sum(axis=-1, keepdims=True))
    tgt = tok[:, 1:]
    nll = -np.take_along_axis(logp, tgt[..., None], axis=-1)[..., 0]
    valid = real[:, 1:]
    n = int(valid.sum())
    return [float((nll * valid).sum() / n), n]

Part 1. Find every bug and write a correct lm_loss with the same signature. Logits can reach the hundreds, so the loss must be computed stably.

Part 2. In the full multi-GPU training run you would not have had this neat reduction. Describe, in order, how you would have localized these failures there: how to make an intermittent failure reproducible, how to tell whether the crash and the bad loss share a cause, what to instrument, and what regression tests you would add.

Constraints: $1 \le B \le 4$, $1 \le T \le 8$, $3 \le V \le 9$, $D \le 8$; numpy is available; outputs are checked with relative tolerance $10^{-6}$.

Examples (with E = [[0,0],[1,0],[0,1],[1,1]], P = [[0,0],[0.5,-0.5],[-0.5,0.5]], Wq = Wk = Wv $= I_2$, Wo = [[1,-1,0.5,0],[0,1,-1,0.5]], pad_id = 0):

  • tokens = [[1,2,3],[3,1,2]] -> about [1.6406, 4]. No padding; the buggy code already gets this one right.
  • tokens = [[1,2,3]] -> about [1.4532, 2]. The buggy code crashes on a batch of one row.
  • tokens = [[1,2,3],[0,2,1]] -> about [1.1087, 3]. The buggy code returns [nan, 4].
Show hint

Which symptoms appear only on some batches, and what property of a batch (its size, its padding) changes from one step to the next?

Relevant roles: ML Engineer, LLM Engineer and Deep Learning Engineer.

Related free problems