Debug a Tiny Causal Transformer and Its Loss
Problem statement
A teammate wrote a tiny one-layer causal transformer language model and its loss in numpy. The code runs without errors, but two unit tests against the reference model fail: the logits for a fixed input and seed do not match, and the one-step training loss does not match either. Their notes say there are four independent bugs: one in the attention mask, one in parameter initialization, one in how the loss lines up predictions with targets, and a fourth one they did not identify.
Reference model (what the code must compute). For a token sequence of length $T$ ($2 \le T \le 16$), vocabulary size $V$ and model width $D$:
- Parameters come from
rng = np.random.default_rng(seed), drawn in this order: token embeddings $(V \times D)$ and position embeddings $(16 \times D)$, both standard normal times 0.02; then $W_q, W_k, W_v, W_o$ $(D \times D)$ and $W_{out}$ $(D \times V)$, each standard normal times $1/\sqrt{D}$. - $x_t = \text{tok}[\text{tokens}_t] + \text{pos}_t$. Single-head attention: $Q = XW_q$, $K = XW_k$, $V' = XW_v$, scores $QK^T/\sqrt{D}$, causal: position $i$ may attend only to positions $j \le i$; softmax over the key positions; output $Y = X + (\text{attn} \, V') W_o$; logits $= Y W_{out}$ $(T \times V)$.
- Loss: mean cross-entropy over positions $t = 0, \dots, T-2$, where the logits at position $t$ predict
tokens[t+1].
Fix the code so that forward_and_loss(tokens, vocab_size, d_model, seed) returns [loss, logits] matching the reference (relative tolerance $10^{-6}$), with logits as a $T \times V$ list of lists. Keep the random draws in the same order. Then explain how you would find bugs like these systematically.
import numpy as np
MAX_LEN = 16
def softmax(z, axis):
z = z - z.max(axis=axis, keepdims=True)
e = np.exp(z)
return e / e.sum(axis=axis, keepdims=True)
def init_params(vocab_size, d_model, seed):
rng = np.random.default_rng(seed)
D = d_model
return {
"tok_emb": rng.standard_normal((vocab_size, D)) * 0.02,
"pos_emb": rng.standard_normal((MAX_LEN, D)) * 0.02,
"Wq": rng.standard_normal((D, D)) / D,
"Wk": rng.standard_normal((D, D)) / D,
"Wv": rng.standard_normal((D, D)) / D,
"Wo": rng.standard_normal((D, D)) / D,
"W_out": rng.standard_normal((D, vocab_size)) / D,
}
def forward(params, tokens):
T = len(tokens)
D = params["Wq"].shape[0]
x = params["tok_emb"][tokens] + params["pos_emb"][:T] # (T, D)
q, k, v = x @ params["Wq"], x @ params["Wk"], x @ params["Wv"]
scores = q @ k.T / np.sqrt(D) # (T, T)
mask = np.tril(np.ones((T, T)), k=-1).astype(bool)
scores = np.where(mask, -1e9, scores)
att = softmax(scores, axis=0)
y = x + (att @ v) @ params["Wo"]
return y @ params["W_out"] # (T, V)
def cross_entropy(logits, targets):
z = logits - logits.max(axis=1, keepdims=True)
logp = z - np.log(np.exp(z).sum(axis=1, keepdims=True))
return float(-logp[np.arange(len(targets)), targets].mean())
def forward_and_loss(tokens, vocab_size, d_model, seed):
tokens = np.asarray(tokens)
params = init_params(vocab_size, d_model, seed)
logits = forward(params, tokens)
loss = cross_entropy(logits[1:], tokens[:-1])
return [loss, logits.tolist()]
Constraints
- $2 \le T \le 16$, $2 \le V \le 40$, $D \in \{2, 4, 8, 12, 16, 32\}$, token ids in $[0, V)$.
- Use numpy only.
Example
forward_and_loss([3, 1, 4, 1, 5, 9, 2, 6], 10, 8, 0)
reference loss: 2.2996939 (close to ln 10 = 2.3026, as expected for a freshly initialized model)Show hint
What invariants should a causal attention layer and a next-token loss satisfy that you could test without the reference outputs?
Relevant roles: ML Engineer, LLM Engineer, Deep Learning Engineer and Research Scientist.
Related free problems
- Debug a Left-Padded Causal LM LossLLMs & GenAIHard
- Multi-Head AttentionMedium
- Masked Self-AttentionMedium