Pre-Norm vs Post-Norm Transformer Block

MediumML Coding

Problem statement

In transformer architectures, the placement of layer normalization relative to the sublayers has a significant impact on training stability and performance. There are two dominant patterns:

  • Post-Norm: The original Transformer architecture applies layer normalization after the residual connection.
  • Pre-Norm: Most modern decoder-only language models apply layer normalization before the sublayer function.

Note on simplification. A real transformer block contains a multi-head self-attention sublayer followed by a feed-forward sublayer. To keep this exercise focused on the placement of normalization and residuals, which is the same regardless of what the sublayer computes, both sublayers here are simplified to plain linear projections z @ W + b. The pre-norm vs. post-norm pattern you implement transfers directly to a real block; just substitute attention for sublayer 1 and an MLP for sublayer 2.

Your Task

Implement a function transformer_block that processes an input through two sublayers (each a simple linear transformation), with residual connections and layer normalization placed according to the specified mode.

The function takes:

  • x: Input array of shape (seq_len, d_model)
  • W1, b1: Weight matrix and bias for the first sublayer (linear transformation)
  • W2, b2: Weight matrix and bias for the second sublayer (linear transformation)
  • gamma1, beta1: Scale and shift parameters for the first layer normalization
  • gamma2, beta2: Scale and shift parameters for the second layer normalization
  • mode: Either "pre_norm" or "post_norm"
  • eps: Small constant for numerical stability in layer normalization (default 1e-5)

Each sublayer is a simple linear projection: sublayer(z) = z @ W + b. (In a real transformer, sublayer 1 would be multi-head self-attention and sublayer 2 a position-wise feed-forward network; both are omitted here so the exercise tests only the norm/residual structure.)

Layer normalization normalizes across the last axis (features) for each token independently, using population variance.

Return the output as a numpy array.

Example

Input:

x = np.array([[1.0, -1.0]])
W1 = np.eye(2) * 0.1, b1 = np.zeros(2)
W2 = np.eye(2) * 0.1, b2 = np.zeros(2)
gamma1 = np.ones(2), beta1 = np.zeros(2)
gamma2 = np.ones(2), beta2 = np.zeros(2)
mode = 'pre_norm'

Output:

[[1.2, -1.2]]

Reasoning: Pre-norm path: First, layer norm is applied to x=[1,-1], yielding approximately [1,-1] (mean=0, var=1). The first sublayer computes 0.1*[1,-1]=[0.1,-0.1]. The residual gives h=[1.1,-1.1]. Layer norm of h again yields approximately [1,-1]. The second sublayer gives [0.1,-0.1], and the final residual produces [1.2,-1.2]. Each sublayer adds to the residual stream, and the values grow. In contrast, post_norm with the same inputs returns [[1.0,-1.0]] because layer normalization after the residual re-normalizes the output back to unit variance each time.

Show hint

Where does the normalization sit relative to the residual addition in each of the two variants?

Relevant roles: ML Engineer, Deep Learning Engineer, LLM Engineer and Research Scientist.

Related free problems