RMSNorm (Root Mean Square Layer Normalization)

EasyML Coding

Problem statement

Implement Root Mean Square Layer Normalization (RMSNorm), the normalization used in many modern transformer language models. Unlike standard Layer Normalization, RMSNorm does not center the activations by subtracting the mean. It rescales each sample by its root mean square only, which is cheaper and works about as well in practice.

Write rmsnorm(x, g, eps=1e-5) where:

  • x: a 2-D NumPy array of shape (batch_size, features) of input activations
  • g: a 1-D NumPy array of shape (features,), the learnable gain
  • eps: a small constant for numerical stability

Normalize each row independently with the RMS computed over the feature dimension, with eps added inside the square root:

$$\hat{x}_i = \frac{x_i}{\sqrt{\frac{1}{n}\sum_{j=1}^{n} x_j^2 + \epsilon}} \cdot g_i.$$

Return an array with the same shape as x.

Example

Input:

x = np.array([[1.0, 2.0, 3.0]])
g = np.array([1.0, 1.0, 1.0])
rmsnorm(x, g)

Output:

[[0.4629, 0.9258, 1.3887]]

Reasoning: the squares are 1, 4 and 9, whose mean is 14/3, about 4.6667. Adding $10^{-5}$ and taking the square root gives an RMS of about 2.1602. Dividing each entry by it gives 0.4629, 0.9258 and 1.3887, and the all-ones gain leaves them unchanged.

Show hint

Which single per-sample statistic measures the typical magnitude of a vector without caring about its mean?

Relevant roles: ML Engineer, LLM Engineer, Deep Learning Engineer, Research Scientist and AI Engineer.

Related free problems