Pre-Norm GPT Transformer Block Forward Pass
Problem statement
Implement the forward pass of a pre-norm Transformer block as used in GPT-style language models.
Given an input tensor x of shape (batch, seq_len, emb_dim) and a dictionary params with the block's weights, compute:
- Attention sub-block (pre-norm + residual):
h = LayerNorm(x)with gammaln1_gamma, betaln1_beta,eps = 1e-5(normalize over the last axis, population variance).h = MultiHeadCausalSelfAttention(h)with weightsW_q,W_k,W_v,W_o(no biases). Split intonum_headsheads, scale scores by $1/\sqrt{d_{head}}$, set future positions (upper triangle) to $-\infty$ before the softmax, then concatenate the heads and project withW_o.x = x + h
- Feed-forward sub-block (pre-norm + residual):
h = LayerNorm(x)withln2_gamma,ln2_beta,eps = 1e-5.h = GELU(h @ W_ff1 + b_ff1) @ W_ff2 + b_ff2, with the tanh GELU approximation $$\text{GELU}(z)=0.5\,z\,\big(1+\tanh\big(\sqrt{2/\pi}\,(z+0.044715\,z^3)\big)\big).$$ The hidden FFN width is4 * emb_dim.x = x + h
Note on the residual stream. Both assignments overwrite x. The x in step 2 is the updated tensor from step 1 (the running residual stream), not the block's original input: x <- x + MHA(LN1(x)), then x <- x + FFN(LN2(x)).
Return the resulting numpy array without rounding (tests round for display). No dropout.
Example
Input:
x = np.array([[[1.0, 2.0], [3.0, 4.0]]]), num_heads = 1
# LayerNorm gammas = 1, betas = 0; W_q = W_k = W_v = W_o = identity (2x2)
# W_ff1, b_ff1, W_ff2, b_ff2 all zeros
Output (rounded to 4 decimals):
[[[0.0, 3.0], [2.0, 5.0]]]
Reasoning: LayerNorm maps both tokens to about $[-1, 1]$. With identity projections and the causal mask, token 0 attends only to itself and token 1 averages two identical vectors, so the attention output is $[-1, 1]$ for both. Adding the residual gives $[0, 3]$ and $[2, 5]$. The zero FFN adds nothing.
Show hint
Which pieces does the block decompose into, and which of them can you write and test as small helpers first?
Relevant roles: ML Engineer, LLM Engineer, Deep Learning Engineer, Research Scientist and AI Engineer.
Related free problems
- Multi-Head AttentionMedium
- Masked Self-AttentionMedium