Pre-Norm GPT Transformer Block Forward Pass

HardML Coding

Reported in interviews at Google

Problem statement

Implement the forward pass of a pre-norm Transformer block as used in GPT-style language models.

Given an input tensor x of shape (batch, seq_len, emb_dim) and a dictionary params with the block's weights, compute:

  1. Attention sub-block (pre-norm + residual):
    • h = LayerNorm(x) with gamma ln1_gamma, beta ln1_beta, eps = 1e-5 (normalize over the last axis, population variance).
    • h = MultiHeadCausalSelfAttention(h) with weights W_q, W_k, W_v, W_o (no biases). Split into num_heads heads, scale scores by $1/\sqrt{d_{head}}$, set future positions (upper triangle) to $-\infty$ before the softmax, then concatenate the heads and project with W_o.
    • x = x + h
  1. Feed-forward sub-block (pre-norm + residual):
    • h = LayerNorm(x) with ln2_gamma, ln2_beta, eps = 1e-5.
    • h = GELU(h @ W_ff1 + b_ff1) @ W_ff2 + b_ff2, with the tanh GELU approximation $$\text{GELU}(z)=0.5\,z\,\big(1+\tanh\big(\sqrt{2/\pi}\,(z+0.044715\,z^3)\big)\big).$$ The hidden FFN width is 4 * emb_dim.
    • x = x + h

Note on the residual stream. Both assignments overwrite x. The x in step 2 is the updated tensor from step 1 (the running residual stream), not the block's original input: x <- x + MHA(LN1(x)), then x <- x + FFN(LN2(x)).

Return the resulting numpy array without rounding (tests round for display). No dropout.

Example

Input:

x = np.array([[[1.0, 2.0], [3.0, 4.0]]]), num_heads = 1
# LayerNorm gammas = 1, betas = 0; W_q = W_k = W_v = W_o = identity (2x2)
# W_ff1, b_ff1, W_ff2, b_ff2 all zeros

Output (rounded to 4 decimals):

[[[0.0, 3.0], [2.0, 5.0]]]

Reasoning: LayerNorm maps both tokens to about $[-1, 1]$. With identity projections and the causal mask, token 0 attends only to itself and token 1 averages two identical vectors, so the attention output is $[-1, 1]$ for both. Adding the residual gives $[0, 3]$ and $[2, 5]$. The zero FFN adds nothing.

Show hint

Which pieces does the block decompose into, and which of them can you write and test as small helpers first?

Relevant roles: ML Engineer, LLM Engineer, Deep Learning Engineer, Research Scientist and AI Engineer.

Related free problems