Rotary Positional Embeddings (RoPE)
Problem statement
Implement Rotary Positional Embeddings (RoPE), which encode position by rotating query and key vectors instead of adding a position vector to them.
Write apply_rope(x, positions, base=10000.0) where:
x: shape(seq_len, d)withdevenpositions: shape(seq_len,), the integer position of each rowbase: the base used to compute the rotation frequencies
Return an array of shape (seq_len, d) with RoPE applied.
RoPE pairs consecutive dimensions $(x_{2i}, x_{2i+1})$ for $i = 0, \ldots, d/2 - 1$ and rotates each pair in its plane. Pair $i$ has frequency $$\theta_i = \text{base}^{-2i/d},$$ and a token at position $m$ rotates pair $i$ by the angle $m\theta_i$: $$x'_{2i} = x_{2i}\cos(m\theta_i) - x_{2i+1}\sin(m\theta_i), \qquad x'_{2i+1} = x_{2i}\sin(m\theta_i) + x_{2i+1}\cos(m\theta_i).$$ Low-index pairs rotate quickly and encode fine-grained position; high-index pairs rotate slowly and encode coarse position.
Example
Input:
x = np.array([[1.0, 0.0], [0.0, 1.0]])
positions = np.array([0, 1])
apply_rope(x, positions, base=10000.0)
Output:
[[1.0, 0.0], [-0.8415, 0.5403]]
Reasoning: with d = 2 there is one pair and $\theta_0 = 1$. At position 0 the angle is 0, so [1.0, 0.0] is unchanged. At position 1 the angle is 1 radian, and (0, 1) becomes $(0 \cdot \cos 1 - 1 \cdot \sin 1,\ 0 \cdot \sin 1 + 1 \cdot \cos 1) = (-0.8415, 0.5403)$.
Show hint
What operation can you apply to a pair of numbers that changes its direction according to position but never changes its length?
Relevant roles: LLM Engineer, ML Engineer, Deep Learning Engineer, Research Scientist and Gen AI Engineer.
Related free problems
- Multi-Head AttentionMedium
- Masked Self-AttentionMedium