Chapter 4: LLM inference
Quantization: rounding error, signal-to-noise and faster decoding
You will be able to estimate the noise that fewer bits add and the decode speed-up they buy. Interviewers ask for both as quick estimates.
What you will learn
- Estimate quantization noise and SNR from bit width
- Estimate decode speedup from reduced weight bytes
Key terms
- bit width b
- the number of bits stored per value; with $b$ bits the grid has $2^b$ ticks
- tick
- one value the grid can store
- range width R
- the width the grid covers, cut into $2^b$ equal steps: 8 for −4 to 4
- step Δ
- the gap between two neighbouring ticks: $R / 2^b$
- rounding error e
- the stored tick minus the true value
Prerequisites
Take Scale, zero point and granularity first. It comes right before this lesson in the course.
Where this lesson picks up: You mapped floats onto an integer grid with a scale and a zero point, and per-channel or per-group scales kept it finer. That left a question open: how much error does a coarser grid add, and what do the saved bytes buy?
What comes next
Next, you will check whether quantizing hurt the model's answers, with KL divergence, top-1 agreement and perplexity.
Next lesson: Did quantizing hurt the model? KL divergence, top-1 agreement and perplexity
More lessons in this part
- Scale, zero point and granularity 22 min
- Did quantizing hurt the model? KL divergence, top-1 agreement and perplexity 16 min
- Choosing a method: weight-only, W8A8, FP8, GPTQ, AWQ, SmoothQuant 21 min
Take the lesson
Short cards, one idea at a time, with quick checks as you go.