RoPE with Position Offset for KV Cache
Problem statement
When generating tokens autoregressively with a KV cache, each new query or key vector sits at an absolute position later than 0. The rotary positional embedding (RoPE) applied to it must therefore use the cos/sin values for its absolute position, not position 0.
Implement apply_rope_with_offset(x, cos, sin, start_pos) that rotates the last dimension of x with a precomputed RoPE table, starting at absolute position start_pos.
Inputs
x: array of shape(batch, n_heads, seq_len, head_dim)(head_dimis even).cos,sin: precomputed tables of shape(max_seq_len, head_dim).start_pos: non-negative integer, the absolute position of the first token inx.
Use the half-rotation convention: split the last dimension into halves x1 = x[..., :head_dim//2] and x2 = x[..., head_dim//2:], build rotated = concat([-x2, x1], axis=-1), and return
out = x * cos_slice + rotated * sin_slice
where cos_slice = cos[start_pos:start_pos+seq_len] and likewise for sin (broadcast over batch and heads). Return the result as a nested Python list (via .tolist()).
Example
Input:
x = [[[[1.0, 2.0, 3.0, 4.0]]]] # shape (1, 1, 1, 4)
cos = [[1,1,1,1], [0,1,0,1], [-1,1,-1,1], [0,1,0,1]]
sin = [[0,0,0,0], [1,0,1,0], [0,0,0,0], [-1,0,-1,0]]
start_pos = 1
Output:
[[[[-3.0, 2.0, 1.0, 4.0]]]]
Reasoning: with head_dim = 4, x1 = [1, 2] and x2 = [3, 4], so rotated = [-3, -4, 1, 2]. Row 1 of the tables gives cos = [0, 1, 0, 1] and sin = [1, 0, 1, 0]. Then out = [0, 2, 0, 4] + [-3, 0, 1, 0] = [-3, 2, 1, 4].
Show hint
Which rows of the precomputed tables correspond to the tokens you are rotating when some tokens are already in the cache?
Relevant roles: LLM Engineer, ML Engineer, Deep Learning Engineer and AI Engineer.
Related free problems
- Multi-Head AttentionMedium
- Masked Self-AttentionMedium