Chapter 4: LLM inference

Quantization: rounding error, signal-to-noise and faster decoding

You will be able to estimate the noise that fewer bits add and the decode speed-up they buy. Interviewers ask for both as quick estimates.

14 minInteractive lessonChapter 4: LLM inferenceLesson 12 of 14

What you will learn

  • Estimate quantization noise and SNR from bit width
  • Estimate decode speedup from reduced weight bytes

Key terms

bit width b
the number of bits stored per value; with $b$ bits the grid has $2^b$ ticks
tick
one value the grid can store
range width R
the width the grid covers, cut into $2^b$ equal steps: 8 for −4 to 4
step Δ
the gap between two neighbouring ticks: $R / 2^b$
rounding error e
the stored tick minus the true value

Prerequisites

Take Scale, zero point and granularity first. It comes right before this lesson in the course.

Where this lesson picks up: You mapped floats onto an integer grid with a scale and a zero point, and per-channel or per-group scales kept it finer. That left a question open: how much error does a coarser grid add, and what do the saved bytes buy?

What comes next

Next, you will check whether quantizing hurt the model's answers, with KL divergence, top-1 agreement and perplexity.

Next lesson: Did quantizing hurt the model? KL divergence, top-1 agreement and perplexity

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.