Chapter 4: LLM inference

Did quantizing hurt the model? KL divergence, top-1 agreement and perplexity

Before a team ships a quantized model, someone must show that its answers stayed good. Interviewers ask how you would check.

16 minInteractive lessonChapter 4: LLM inferenceLesson 13 of 14

What you will learn

  • Check quality with KL divergence, top-1 agreement and perplexity delta

Key terms

top word
the next word a model gives the highest chance
top-1 agreement
the share of positions where both models pick the same top word
KL divergence
how far one set of chances moved from another; 0 when equal
ln
the natural log: the power of e ≈ 2.718 that gives the number
perplexity
exp of the average loss on held-out text; lower is better

Prerequisites

Take Quantization: rounding error, signal-to-noise and faster decoding first. It comes right before this lesson in the course.

Where this lesson picks up: Last lesson, fewer bits made decoding faster, but each weight gained a small rounding error. How do we check that the smaller model still answers like the original?

What comes next

Next, you will choose a method for weights and activations whose outliers make them hard to quantize.

Next lesson: Choosing a method: weight-only, W8A8, FP8, GPTQ, AWQ, SmoothQuant

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.