Chapter 4: LLM inference
Did quantizing hurt the model? KL divergence, top-1 agreement and perplexity
Before a team ships a quantized model, someone must show that its answers stayed good. Interviewers ask how you would check.
What you will learn
- Check quality with KL divergence, top-1 agreement and perplexity delta
Key terms
- top word
- the next word a model gives the highest chance
- top-1 agreement
- the share of positions where both models pick the same top word
- KL divergence
- how far one set of chances moved from another; 0 when equal
- ln
- the natural log: the power of e ≈ 2.718 that gives the number
- perplexity
- exp of the average loss on held-out text; lower is better
Prerequisites
Take Quantization: rounding error, signal-to-noise and faster decoding first. It comes right before this lesson in the course.
Where this lesson picks up: Last lesson, fewer bits made decoding faster, but each weight gained a small rounding error. How do we check that the smaller model still answers like the original?
What comes next
Next, you will choose a method for weights and activations whose outliers make them hard to quantize.
Next lesson: Choosing a method: weight-only, W8A8, FP8, GPTQ, AWQ, SmoothQuant
More lessons in this part
- Scale, zero point and granularity 22 min
- Quantization: rounding error, signal-to-noise and faster decoding 14 min
- Choosing a method: weight-only, W8A8, FP8, GPTQ, AWQ, SmoothQuant 21 min
Take the lesson
Short cards, one idea at a time, with quick checks as you go.