INT8 KV-Cache Quantize and Dequantize

EasyML Coding

Problem statement

Implement quantize_kv_int8(x): quantize a KV-cache tensor to symmetric INT8 along the last axis and reconstruct the dequantized values.

Input

  • x: float array of shape (..., d). The last axis is the head dimension (or a contiguous group). Typical layouts are (batch, seq, n_kv_heads, head_dim) or (seq, n_kv_heads, head_dim).

Per-group map (one scale per last-axis vector)

$$s = \frac{\max |x|}{127}, \qquad q = \mathrm{clip}\big(\mathrm{round}(x / s),\, -127,\, 127\big), \qquad \hat x = q \cdot s$$

If $\max|x| < 10^{-12}$ on a vector, use $s = 1$ so a zero group does not divide by zero. round is round-half-to-even, exactly as numpy.rint does. q is stored as int8.

Return a tuple (q, scale, x_hat):

  • q: same shape as x, dtype int8
  • scale: same shape as x except the last axis has size 1, float
  • x_hat: dequantized reconstruction, same shape as x

Example

Input:

x = [[1.27, -2.54]]

Output:

q = [[64, -127]] (int8), scale = [[0.02]], x_hat = [[1.28, -2.54]]

Reasoning: The last-axis absmax is 2.54, so the scale is 2.54/127 = 0.02. Rounding 1.27/0.02 gives 64 and -2.54/0.02 saturates at -127. Multiplying back recovers 1.28 and -2.54.

Show hint

With a symmetric grid and no zero-point, what single number per vector maps the largest magnitude onto the top integer?

Relevant roles: LLM Engineer, Deep Learning Engineer, MLOps Engineer and ML Engineer.

Related free problems