RMSNorm (Root Mean Square Layer Normalization)
Problem statement
Implement Root Mean Square Layer Normalization (RMSNorm), the normalization used in many modern transformer language models. Unlike standard Layer Normalization, RMSNorm does not center the activations by subtracting the mean. It rescales each sample by its root mean square only, which is cheaper and works about as well in practice.
Write rmsnorm(x, g, eps=1e-5) where:
x: a 2-D NumPy array of shape(batch_size, features)of input activationsg: a 1-D NumPy array of shape(features,), the learnable gaineps: a small constant for numerical stability
Normalize each row independently with the RMS computed over the feature dimension, with eps added inside the square root:
$$\hat{x}_i = \frac{x_i}{\sqrt{\frac{1}{n}\sum_{j=1}^{n} x_j^2 + \epsilon}} \cdot g_i.$$
Return an array with the same shape as x.
Example
Input:
x = np.array([[1.0, 2.0, 3.0]])
g = np.array([1.0, 1.0, 1.0])
rmsnorm(x, g)
Output:
[[0.4629, 0.9258, 1.3887]]
Reasoning: the squares are 1, 4 and 9, whose mean is 14/3, about 4.6667. Adding $10^{-5}$ and taking the square root gives an RMS of about 2.1602. Dividing each entry by it gives 0.4629, 0.9258 and 1.3887, and the all-ones gain leaves them unchanged.
Show hint
Which single per-sample statistic measures the typical magnitude of a vector without caring about its mean?
Relevant roles: ML Engineer, LLM Engineer, Deep Learning Engineer, Research Scientist and AI Engineer.
Related free problems
- Multi-Head AttentionMedium
- Masked Self-AttentionMedium