Decoder-Only Transformer Forward Pass from Scratch
Problem statement
With no scaffolding, implement the forward pass of a small decoder-only language model in numpy and explain the tensor shapes as you go. Given a batch of token-id sequences, it returns next-token logits for every position, and the logits at position $t$ may depend only on tokens $0..t$.
The architecture, with $B$ sequences of length $T$, width $D$, $H$ heads of size $d_h = D / H$, vocabulary $V$:
- Embeddings:
x = tok_emb[tokens] + pos_emb[:T](learned absolute positions), shape $(B, T, D)$. - A stack of decoder blocks, each pre-norm with residual connections:
x = x + Attn(LN1(x)): causal multi-head self-attention.q = z @ w_q + b_q, and likewisek,v(weights $D \times D$, biases of length $D$). Head $h$ uses columns $h d_h$ to $(h+1) d_h - 1$. Scores are $q k^\top / \sqrt{d_h}$; position $i$ may attend only to positions $j \le i$, so future scores are excluded before a softmax over keys. Head outputs are concatenated in head order and mapped byw_o,b_o.x = x + FFN(LN2(x))withFFN(z) = relu(z @ w_1 + b_1) @ w_2 + b_2.
- Output: a final LayerNorm, then
logits = LN_f(x) @ w_out, shape $(B, T, V)$.
Every LayerNorm normalizes over the last axis using the population variance and $\epsilon = 10^{-5}$, then applies g * z_hat + b. Attention must be written by hand (no library attention module), and all heads should be computed with batched matrix products rather than a Python loop over heads.
def decoder_logits(tokens: list, params: dict, n_heads: int) -> list:
# tokens: [B][T] ints
# params: tok_emb [V][D], pos_emb [P][D] (P >= T), lnf_g, lnf_b [D], w_out [D][V],
# blocks: list of dicts with ln1_g, ln1_b, w_q, b_q, w_k, b_k, w_v, b_v,
# w_o, b_o, ln2_g, ln2_b, w_1 [D][F], b_1 [F], w_2 [F][D], b_2
# returns logits as a nested list [B][T][V]
...
Constraints
- $1 \le B \le 3$, $1 \le T \le 8$, $D \le 16$, up to 3 blocks; $D$ is divisible by
n_heads. - Outputs are compared with a relative tolerance of $10^{-6}$.
Example 1
$D = 2$, one head, $V = 3$, one block whose weights and biases are all zero (so the block adds nothing to the residual stream), unit gains and zero shifts in every LayerNorm.
tok_emb = [[1, 0], [0, 1], [2, 2]] pos_emb = [[0, 0], [0.5, 0], [0, 0]]
w_out = [[1, 0, 1], [0, 1, 1]] tokens = [[0, 1, 2]]
output: [[[0.99998, -0.99998, 0.0], [-0.99992, 0.99992, 0.0], [0.0, 0.0, 0.0]]]
Position 1 sees $x = (0.5, 1)$, which the final LayerNorm maps to about $(-1, 1)$; position 2 sees $(2, 2)$, which has zero variance and normalizes to $(0, 0)$.
Example 2 (property check). With random weights, changing only the last token of each sequence must leave the logits at every earlier position unchanged.
Show hint
Before writing code, what is the shape of the activations entering each sub-layer and the shape of the attention score tensor?
Relevant roles: ML Engineer, Deep Learning Engineer, LLM Engineer and AI Engineer.
Related free problems
- Multi-Head AttentionMedium
- Masked Self-AttentionMedium