Rotary Positional Embeddings (RoPE)

MediumML Coding

Problem statement

Implement Rotary Positional Embeddings (RoPE), which encode position by rotating query and key vectors instead of adding a position vector to them.

Write apply_rope(x, positions, base=10000.0) where:

  • x: shape (seq_len, d) with d even
  • positions: shape (seq_len,), the integer position of each row
  • base: the base used to compute the rotation frequencies

Return an array of shape (seq_len, d) with RoPE applied.

RoPE pairs consecutive dimensions $(x_{2i}, x_{2i+1})$ for $i = 0, \ldots, d/2 - 1$ and rotates each pair in its plane. Pair $i$ has frequency $$\theta_i = \text{base}^{-2i/d},$$ and a token at position $m$ rotates pair $i$ by the angle $m\theta_i$: $$x'_{2i} = x_{2i}\cos(m\theta_i) - x_{2i+1}\sin(m\theta_i), \qquad x'_{2i+1} = x_{2i}\sin(m\theta_i) + x_{2i+1}\cos(m\theta_i).$$ Low-index pairs rotate quickly and encode fine-grained position; high-index pairs rotate slowly and encode coarse position.

Example

Input:

x = np.array([[1.0, 0.0], [0.0, 1.0]])
positions = np.array([0, 1])
apply_rope(x, positions, base=10000.0)

Output:

[[1.0, 0.0], [-0.8415, 0.5403]]

Reasoning: with d = 2 there is one pair and $\theta_0 = 1$. At position 0 the angle is 0, so [1.0, 0.0] is unchanged. At position 1 the angle is 1 radian, and (0, 1) becomes $(0 \cdot \cos 1 - 1 \cdot \sin 1,\ 0 \cdot \sin 1 + 1 \cdot \cos 1) = (-0.8415, 0.5403)$.

Show hint

What operation can you apply to a pair of numbers that changes its direction according to position but never changes its length?

Relevant roles: LLM Engineer, ML Engineer, Deep Learning Engineer, Research Scientist and Gen AI Engineer.

Related free problems