RoPE with Position Offset for KV Cache

MediumML Coding

Problem statement

When generating tokens autoregressively with a KV cache, each new query or key vector sits at an absolute position later than 0. The rotary positional embedding (RoPE) applied to it must therefore use the cos/sin values for its absolute position, not position 0.

Implement apply_rope_with_offset(x, cos, sin, start_pos) that rotates the last dimension of x with a precomputed RoPE table, starting at absolute position start_pos.

Inputs

  • x: array of shape (batch, n_heads, seq_len, head_dim) (head_dim is even).
  • cos, sin: precomputed tables of shape (max_seq_len, head_dim).
  • start_pos: non-negative integer, the absolute position of the first token in x.

Use the half-rotation convention: split the last dimension into halves x1 = x[..., :head_dim//2] and x2 = x[..., head_dim//2:], build rotated = concat([-x2, x1], axis=-1), and return

out = x * cos_slice + rotated * sin_slice

where cos_slice = cos[start_pos:start_pos+seq_len] and likewise for sin (broadcast over batch and heads). Return the result as a nested Python list (via .tolist()).

Example

Input:

x = [[[[1.0, 2.0, 3.0, 4.0]]]]          # shape (1, 1, 1, 4)
cos = [[1,1,1,1], [0,1,0,1], [-1,1,-1,1], [0,1,0,1]]
sin = [[0,0,0,0], [1,0,1,0], [0,0,0,0], [-1,0,-1,0]]
start_pos = 1

Output:

[[[[-3.0, 2.0, 1.0, 4.0]]]]

Reasoning: with head_dim = 4, x1 = [1, 2] and x2 = [3, 4], so rotated = [-3, -4, 1, 2]. Row 1 of the tables gives cos = [0, 1, 0, 1] and sin = [1, 0, 1, 0]. Then out = [0, 2, 0, 4] + [-3, 0, 1, 0] = [-3, 2, 1, 4].

Show hint

Which rows of the precomputed tables correspond to the tokens you are rotating when some tokens are already in the cache?

Relevant roles: LLM Engineer, ML Engineer, Deep Learning Engineer and AI Engineer.

Related free problems