Chapter 4: LLM inference
Scale, zero point and granularity
A model can store short integer labels instead of full floating-point weights. In interviews, explain the ruler that gives those labels meaning.
What you will learn
- Compute symmetric and asymmetric scale and zero point
- Quantize and dequantize per channel
- Explain why per-group quantization helps at 4 bits
Key terms
- Quantization
- Replace model numbers with stored integer codes.
- Integer code
- A whole-number label that needs a stored ruler.
- Reconstruction
- Convert stored labels back to approximate numbers.
Prerequisites
Take Speculative decoding: the accept rule and expected speedup first. It comes right before this lesson in the course.
Where this lesson picks up: You reduced token work with a draft model and a check. Now you reduce the storage for model numbers; no draft-model maths is needed.
What comes next
Measure reconstruction error, memory savings and changes in model outputs.
Next lesson: Quantization: rounding error, signal-to-noise and faster decoding
More lessons in this part
- Quantization: rounding error, signal-to-noise and faster decoding 14 min
- Did quantizing hurt the model? KL divergence, top-1 agreement and perplexity 16 min
- Choosing a method: weight-only, W8A8, FP8, GPTQ, AWQ, SmoothQuant 21 min
Take the lesson
Short cards, one idea at a time, with quick checks as you go.