Transformer Encoder Layer
Problem statement
Implement a complete Transformer encoder layer forward pass using only NumPy. The layer follows the original post-norm Transformer and applies four sub-operations in sequence:
- Multi-head self-attention: project the input into queries, keys and values, split them into heads, run scaled dot-product attention per head, concatenate the heads and apply an output projection.
- Add and norm (first): add the sub-layer input to its output (residual), then apply layer normalization.
- Position-wise feed-forward network: two linear layers with a ReLU in between.
- Add and norm (second): another residual connection followed by layer normalization.
Write transformer_encoder_layer(X, weights, num_heads, eps=1e-5):
X: input of shape(batch_size, seq_len, d_model).weights: a dict with'W_q','W_k','W_v': projections, each(d_model, d_model);'W_o': output projection(d_model, d_model);'W1','b1': first FFN weight(d_model, d_ff)and bias(d_ff,);'W2','b2': second FFN weight(d_ff, d_model)and bias(d_model,);'gamma1','beta1','gamma2','beta2': layer-norm parameters, each(d_model,).
num_heads: number of heads (dividesd_modelevenly).eps: layer-norm epsilon.
Return an array of shape (batch_size, seq_len, d_model). Use a numerically stable softmax (subtract the max before exponentiating). Layer normalization normalizes over the last dimension with population variance (denominator $N$, not $N-1$).
Example
Input:
X = np.array([[[1.0, 0.0, -1.0, 0.0]]])
# all weight matrices and biases are zero, gamma = 1, beta = 0, d_model = 4, num_heads = 2
Output:
[[[1.4142, 0.0, -1.4142, 0.0]]]
Reasoning: with zero weights, $Q = K = V = 0$, so the attention output is 0 and the first residual gives $X$ back. Layer norm on $[1, 0, -1, 0]$ uses mean 0 and variance 0.5, giving $[1.4142, 0, -1.4142, 0]$. The FFN outputs 0, so the second residual leaves this vector unchanged; it already has mean 0 and variance 1, so the second layer norm returns $[1.4142, 0, -1.4142, 0]$.
Show hint
How can a single projection of width d_model be viewed as several smaller attention heads working side by side?
Relevant roles: ML Engineer, LLM Engineer, Deep Learning Engineer, Research Scientist and AI Engineer.