Transformer Encoder Layer

HardLLMs & GenAI

Problem statement

Implement a complete Transformer encoder layer forward pass using only NumPy. The layer follows the original post-norm Transformer and applies four sub-operations in sequence:

  1. Multi-head self-attention: project the input into queries, keys and values, split them into heads, run scaled dot-product attention per head, concatenate the heads and apply an output projection.
  2. Add and norm (first): add the sub-layer input to its output (residual), then apply layer normalization.
  3. Position-wise feed-forward network: two linear layers with a ReLU in between.
  4. Add and norm (second): another residual connection followed by layer normalization.

Write transformer_encoder_layer(X, weights, num_heads, eps=1e-5):

  • X: input of shape (batch_size, seq_len, d_model).
  • weights: a dict with
    • 'W_q', 'W_k', 'W_v': projections, each (d_model, d_model);
    • 'W_o': output projection (d_model, d_model);
    • 'W1', 'b1': first FFN weight (d_model, d_ff) and bias (d_ff,);
    • 'W2', 'b2': second FFN weight (d_ff, d_model) and bias (d_model,);
    • 'gamma1', 'beta1', 'gamma2', 'beta2': layer-norm parameters, each (d_model,).
  • num_heads: number of heads (divides d_model evenly).
  • eps: layer-norm epsilon.

Return an array of shape (batch_size, seq_len, d_model). Use a numerically stable softmax (subtract the max before exponentiating). Layer normalization normalizes over the last dimension with population variance (denominator $N$, not $N-1$).

Example

Input:

X = np.array([[[1.0, 0.0, -1.0, 0.0]]])
# all weight matrices and biases are zero, gamma = 1, beta = 0, d_model = 4, num_heads = 2

Output:

[[[1.4142, 0.0, -1.4142, 0.0]]]

Reasoning: with zero weights, $Q = K = V = 0$, so the attention output is 0 and the first residual gives $X$ back. Layer norm on $[1, 0, -1, 0]$ uses mean 0 and variance 0.5, giving $[1.4142, 0, -1.4142, 0]$. The FFN outputs 0, so the second residual leaves this vector unchanged; it already has mean 0 and variance 1, so the second layer norm returns $[1.4142, 0, -1.4142, 0]$.

Show hint

How can a single projection of width d_model be viewed as several smaller attention heads working side by side?

Relevant roles: ML Engineer, LLM Engineer, Deep Learning Engineer, Research Scientist and AI Engineer.

Related free problems