Layer Normalization for Sequence Data

EasyML Coding

Problem statement

Implement layer_normalization(X, gamma, beta, epsilon=1e-5) for a batch of sequences.

  • X has shape (batch_size, seq_len, d_model).
  • For every token (every (batch, position) pair), compute the mean $\mu$ and the population variance $\sigma^2$ over the feature dimension (the last axis).
  • Normalize: $\hat x = (x - \mu)/\sqrt{\sigma^2 + \epsilon}$.
  • Scale and shift: $y = \gamma \hat x + \beta$, where gamma and beta have shape (1, 1, d_model) and broadcast over batch and sequence.

Return an array with the same shape as X.

Example

Input:

np.random.seed(42)
X = np.random.randn(2, 2, 3)
gamma = np.ones(3).reshape(1, 1, -1)
beta = np.zeros(3).reshape(1, 1, -1)
print(np.round(layer_normalization(X, gamma, beta), 3))

Output:

[[[ 0.474 -1.391  0.917]
  [ 1.414 -0.707 -0.707]]

 [[ 1.132  0.168 -1.3  ]
  [ 1.414 -0.705 -0.71 ]]]

Reasoning: Each row of 3 features is normalized to mean 0 and variance close to 1 independently of every other token; with $\gamma = 1$ and $\beta = 0$ the normalized values are returned unchanged.

Show hint

Over which dimension should the statistics be computed so that each token is normalized independently of every other token and of the batch?

Relevant roles: ML Engineer, Deep Learning Engineer, LLM Engineer, Research Scientist and AI Engineer.

Related free problems