Layer Normalization for Sequence Data
Problem statement
Implement layer_normalization(X, gamma, beta, epsilon=1e-5) for a batch of sequences.
Xhas shape(batch_size, seq_len, d_model).- For every token (every
(batch, position)pair), compute the mean $\mu$ and the population variance $\sigma^2$ over the feature dimension (the last axis). - Normalize: $\hat x = (x - \mu)/\sqrt{\sigma^2 + \epsilon}$.
- Scale and shift: $y = \gamma \hat x + \beta$, where
gammaandbetahave shape(1, 1, d_model)and broadcast over batch and sequence.
Return an array with the same shape as X.
Example
Input:
np.random.seed(42)
X = np.random.randn(2, 2, 3)
gamma = np.ones(3).reshape(1, 1, -1)
beta = np.zeros(3).reshape(1, 1, -1)
print(np.round(layer_normalization(X, gamma, beta), 3))
Output:
[[[ 0.474 -1.391 0.917]
[ 1.414 -0.707 -0.707]]
[[ 1.132 0.168 -1.3 ]
[ 1.414 -0.705 -0.71 ]]]
Reasoning: Each row of 3 features is normalized to mean 0 and variance close to 1 independently of every other token; with $\gamma = 1$ and $\beta = 0$ the normalized values are returned unchanged.
Show hint
Over which dimension should the statistics be computed so that each token is normalized independently of every other token and of the batch?
Relevant roles: ML Engineer, Deep Learning Engineer, LLM Engineer, Research Scientist and AI Engineer.
Related free problems
- Multi-Head AttentionMedium
- Masked Self-AttentionMedium