Pre-Norm vs Post-Norm Transformer Block
Problem statement
In transformer architectures, the placement of layer normalization relative to the sublayers has a significant impact on training stability and performance. There are two dominant patterns:
- Post-Norm: The original Transformer architecture applies layer normalization after the residual connection.
- Pre-Norm: Most modern decoder-only language models apply layer normalization before the sublayer function.
Note on simplification. A real transformer block contains a multi-head self-attention sublayer followed by a feed-forward sublayer. To keep this exercise focused on the placement of normalization and residuals, which is the same regardless of what the sublayer computes, both sublayers here are simplified to plain linear projections z @ W + b. The pre-norm vs. post-norm pattern you implement transfers directly to a real block; just substitute attention for sublayer 1 and an MLP for sublayer 2.
Your Task
Implement a function transformer_block that processes an input through two sublayers (each a simple linear transformation), with residual connections and layer normalization placed according to the specified mode.
The function takes:
x: Input array of shape(seq_len, d_model)W1, b1: Weight matrix and bias for the first sublayer (linear transformation)W2, b2: Weight matrix and bias for the second sublayer (linear transformation)gamma1, beta1: Scale and shift parameters for the first layer normalizationgamma2, beta2: Scale and shift parameters for the second layer normalizationmode: Either"pre_norm"or"post_norm"eps: Small constant for numerical stability in layer normalization (default1e-5)
Each sublayer is a simple linear projection: sublayer(z) = z @ W + b. (In a real transformer, sublayer 1 would be multi-head self-attention and sublayer 2 a position-wise feed-forward network; both are omitted here so the exercise tests only the norm/residual structure.)
Layer normalization normalizes across the last axis (features) for each token independently, using population variance.
Return the output as a numpy array.
Example
Input:
x = np.array([[1.0, -1.0]])
W1 = np.eye(2) * 0.1, b1 = np.zeros(2)
W2 = np.eye(2) * 0.1, b2 = np.zeros(2)
gamma1 = np.ones(2), beta1 = np.zeros(2)
gamma2 = np.ones(2), beta2 = np.zeros(2)
mode = 'pre_norm'
Output:
[[1.2, -1.2]]
Reasoning: Pre-norm path: First, layer norm is applied to x=[1,-1], yielding approximately [1,-1] (mean=0, var=1). The first sublayer computes 0.1*[1,-1]=[0.1,-0.1]. The residual gives h=[1.1,-1.1]. Layer norm of h again yields approximately [1,-1]. The second sublayer gives [0.1,-0.1], and the final residual produces [1.2,-1.2]. Each sublayer adds to the residual stream, and the values grow. In contrast, post_norm with the same inputs returns [[1.0,-1.0]] because layer normalization after the residual re-normalizes the output back to unit variance each time.
Show hint
Where does the normalization sit relative to the residual addition in each of the two variants?
Relevant roles: ML Engineer, Deep Learning Engineer, LLM Engineer and Research Scientist.
Related free problems
- Transformer Encoder LayerLLMs & GenAIHard
- Multi-Head AttentionMedium
- Masked Self-AttentionMedium