Debug a Left-Padded Causal LM Loss
Problem statement
You inherit the training code of a small decoder-only language model. Training is unreliable: every so often a step crashes with a shape error, most steps report a NaN loss, and on the few batches that do produce a number the loss looks plausible but the model learns badly. To isolate the problem, a teammate reduced the model to a single layer with one attention head and wrote the numpy loss below. It reproduces all three symptoms.
Intended computation. Batches are left-padded: each row of tokens ($B \times T$ integer ids) is some number of pad_id entries followed by the real tokens (a row may be all padding). E is the $V \times D$ token embedding, P the $T_{max} \times D$ position embedding ($T_{max} \ge T$), Wq, Wk, Wv are $D \times D$ and Wo is $D \times V$.
- Positions count real tokens only: the first real token of every row has position 0, the next position 1, and so on (pad slots use position 0; their value never matters).
- $x_t = E[\text{tok}_t] + P[\text{pos}_t]$; $q_t, k_t, v_t = x_t W_q, x_t W_k, x_t W_v$.
- Query $t$ may attend to key $j$ only if $j \le t$ and token $j$ is real; the weights are a softmax of $q_t \cdot k_j / \sqrt{D}$ over the allowed keys. A query with no allowed key gets zero attention output.
- $h_t = x_t + \sum_j a_{tj} v_j$ and $\text{logits}_t = h_t W_o$.
- Position $t$ predicts token $t+1$. A target counts only if both token $t$ and token $t+1$ are real. The loss is the mean of $-\log \mathrm{softmax}(\text{logits}_t)[\text{tok}_{t+1}]$ over the counted targets.
- Return
[mean_loss, n_targets], or[0.0, 0]if the batch has no counted target.
The buggy code:
import numpy as np
def lm_loss(tokens, pad_id, E, P, Wq, Wk, Wv, Wo):
tok = np.array(tokens).squeeze()
E, P, Wq, Wk, Wv, Wo = (np.asarray(a, dtype=float) for a in (E, P, Wq, Wk, Wv, Wo))
B, T = tok.shape
D = E.shape[1]
real = tok != pad_id
pos = np.broadcast_to(np.arange(T), (B, T))
X = E[tok] + P[pos]
Q, K, V = X @ Wq, X @ Wk, X @ Wv
scores = Q @ K.transpose(0, 2, 1) / np.sqrt(D)
causal = np.tril(np.ones((T, T), dtype=bool))
allowed = causal[None, :, :] & real[:, None, :]
masked = np.where(allowed, scores, -np.inf)
e = np.exp(masked - masked.max(axis=-1, keepdims=True))
w = e / e.sum(axis=-1, keepdims=True)
H = X + w @ V
logits = H @ Wo
z = logits[:, :-1, :]
logp = z - np.log(np.exp(z).sum(axis=-1, keepdims=True))
tgt = tok[:, 1:]
nll = -np.take_along_axis(logp, tgt[..., None], axis=-1)[..., 0]
valid = real[:, 1:]
n = int(valid.sum())
return [float((nll * valid).sum() / n), n]
Part 1. Find every bug and write a correct lm_loss with the same signature. Logits can reach the hundreds, so the loss must be computed stably.
Part 2. In the full multi-GPU training run you would not have had this neat reduction. Describe, in order, how you would have localized these failures there: how to make an intermittent failure reproducible, how to tell whether the crash and the bad loss share a cause, what to instrument, and what regression tests you would add.
Constraints: $1 \le B \le 4$, $1 \le T \le 8$, $3 \le V \le 9$, $D \le 8$; numpy is available; outputs are checked with relative tolerance $10^{-6}$.
Examples (with E = [[0,0],[1,0],[0,1],[1,1]], P = [[0,0],[0.5,-0.5],[-0.5,0.5]], Wq = Wk = Wv $= I_2$, Wo = [[1,-1,0.5,0],[0,1,-1,0.5]], pad_id = 0):
tokens = [[1,2,3],[3,1,2]]-> about[1.6406, 4]. No padding; the buggy code already gets this one right.tokens = [[1,2,3]]-> about[1.4532, 2]. The buggy code crashes on a batch of one row.tokens = [[1,2,3],[0,2,1]]-> about[1.1087, 3]. The buggy code returns[nan, 4].
Show hint
Which symptoms appear only on some batches, and what property of a batch (its size, its padding) changes from one step to the next?
Relevant roles: ML Engineer, LLM Engineer and Deep Learning Engineer.
Related free problems
- Debug a Tiny Causal Transformer and Its LossML CodingHard
- Debugging an In-Batch Contrastive LossML CodingHard