Debug a Tiny Causal Transformer and Its Loss

HardML Coding

Reported in interviews at OpenAI

Problem statement

A teammate wrote a tiny one-layer causal transformer language model and its loss in numpy. The code runs without errors, but two unit tests against the reference model fail: the logits for a fixed input and seed do not match, and the one-step training loss does not match either. Their notes say there are four independent bugs: one in the attention mask, one in parameter initialization, one in how the loss lines up predictions with targets, and a fourth one they did not identify.

Reference model (what the code must compute). For a token sequence of length $T$ ($2 \le T \le 16$), vocabulary size $V$ and model width $D$:

  1. Parameters come from rng = np.random.default_rng(seed), drawn in this order: token embeddings $(V \times D)$ and position embeddings $(16 \times D)$, both standard normal times 0.02; then $W_q, W_k, W_v, W_o$ $(D \times D)$ and $W_{out}$ $(D \times V)$, each standard normal times $1/\sqrt{D}$.
  2. $x_t = \text{tok}[\text{tokens}_t] + \text{pos}_t$. Single-head attention: $Q = XW_q$, $K = XW_k$, $V' = XW_v$, scores $QK^T/\sqrt{D}$, causal: position $i$ may attend only to positions $j \le i$; softmax over the key positions; output $Y = X + (\text{attn} \, V') W_o$; logits $= Y W_{out}$ $(T \times V)$.
  3. Loss: mean cross-entropy over positions $t = 0, \dots, T-2$, where the logits at position $t$ predict tokens[t+1].

Fix the code so that forward_and_loss(tokens, vocab_size, d_model, seed) returns [loss, logits] matching the reference (relative tolerance $10^{-6}$), with logits as a $T \times V$ list of lists. Keep the random draws in the same order. Then explain how you would find bugs like these systematically.

import numpy as np

MAX_LEN = 16


def softmax(z, axis):
    z = z - z.max(axis=axis, keepdims=True)
    e = np.exp(z)
    return e / e.sum(axis=axis, keepdims=True)


def init_params(vocab_size, d_model, seed):
    rng = np.random.default_rng(seed)
    D = d_model
    return {
        "tok_emb": rng.standard_normal((vocab_size, D)) * 0.02,
        "pos_emb": rng.standard_normal((MAX_LEN, D)) * 0.02,
        "Wq": rng.standard_normal((D, D)) / D,
        "Wk": rng.standard_normal((D, D)) / D,
        "Wv": rng.standard_normal((D, D)) / D,
        "Wo": rng.standard_normal((D, D)) / D,
        "W_out": rng.standard_normal((D, vocab_size)) / D,
    }


def forward(params, tokens):
    T = len(tokens)
    D = params["Wq"].shape[0]
    x = params["tok_emb"][tokens] + params["pos_emb"][:T]          # (T, D)
    q, k, v = x @ params["Wq"], x @ params["Wk"], x @ params["Wv"]
    scores = q @ k.T / np.sqrt(D)                                    # (T, T)
    mask = np.tril(np.ones((T, T)), k=-1).astype(bool)
    scores = np.where(mask, -1e9, scores)
    att = softmax(scores, axis=0)
    y = x + (att @ v) @ params["Wo"]
    return y @ params["W_out"]                                       # (T, V)


def cross_entropy(logits, targets):
    z = logits - logits.max(axis=1, keepdims=True)
    logp = z - np.log(np.exp(z).sum(axis=1, keepdims=True))
    return float(-logp[np.arange(len(targets)), targets].mean())


def forward_and_loss(tokens, vocab_size, d_model, seed):
    tokens = np.asarray(tokens)
    params = init_params(vocab_size, d_model, seed)
    logits = forward(params, tokens)
    loss = cross_entropy(logits[1:], tokens[:-1])
    return [loss, logits.tolist()]

Constraints

  • $2 \le T \le 16$, $2 \le V \le 40$, $D \in \{2, 4, 8, 12, 16, 32\}$, token ids in $[0, V)$.
  • Use numpy only.

Example

forward_and_loss([3, 1, 4, 1, 5, 9, 2, 6], 10, 8, 0)
reference loss: 2.2996939 (close to ln 10 = 2.3026, as expected for a freshly initialized model)
Show hint

What invariants should a causal attention layer and a next-token loss satisfy that you could test without the reference outputs?

Relevant roles: ML Engineer, LLM Engineer, Deep Learning Engineer and Research Scientist.

Related free problems