INT8 KV-Cache Quantize and Dequantize
Problem statement
Implement quantize_kv_int8(x): quantize a KV-cache tensor to symmetric INT8 along the last axis and reconstruct the dequantized values.
Input
x: float array of shape(..., d). The last axis is the head dimension (or a contiguous group). Typical layouts are(batch, seq, n_kv_heads, head_dim)or(seq, n_kv_heads, head_dim).
Per-group map (one scale per last-axis vector)
$$s = \frac{\max |x|}{127}, \qquad q = \mathrm{clip}\big(\mathrm{round}(x / s),\, -127,\, 127\big), \qquad \hat x = q \cdot s$$
If $\max|x| < 10^{-12}$ on a vector, use $s = 1$ so a zero group does not divide by zero. round is round-half-to-even, exactly as numpy.rint does. q is stored as int8.
Return a tuple (q, scale, x_hat):
q: same shape asx, dtypeint8scale: same shape asxexcept the last axis has size1, floatx_hat: dequantized reconstruction, same shape asx
Example
Input:
x = [[1.27, -2.54]]
Output:
q = [[64, -127]] (int8), scale = [[0.02]], x_hat = [[1.28, -2.54]]
Reasoning: The last-axis absmax is 2.54, so the scale is 2.54/127 = 0.02. Rounding 1.27/0.02 gives 64 and -2.54/0.02 saturates at -127. Multiplying back recovers 1.28 and -2.54.
Show hint
With a symmetric grid and no zero-point, what single number per vector maps the largest magnitude onto the top integer?
Relevant roles: LLM Engineer, Deep Learning Engineer, MLOps Engineer and ML Engineer.
Related free problems
- Multi-Head AttentionMedium
- Masked Self-AttentionMedium