Chapter 4: LLM inference
Choosing a method: weight-only, W8A8, FP8, GPTQ, AWQ, SmoothQuant
You will be able to pick weight-only INT4, W8A8 or FP8 from what limits a deployment's speed, and explain how GPTQ and SmoothQuant work. Interviewers ask this to see if you choose with numbers and measurements, not by habit.
What you will learn
- Explain why activation outliers break INT8 and how SmoothQuant fixes them
- Explain GPTQ's error compensation
- Choose weight-only vs W8A8 vs FP8 for a latency- or throughput-bound deployment
Key terms
- decode
- the step that makes each new token, one at a time
- prefill
- the step that reads the whole prompt at once
- activation
- a temporary number passed between layers
- W8A8
- 8-bit weights and 8-bit activations
- KV cache
- stored keys and values that grow with batch and context
Prerequisites
Take Did quantizing hurt the model? KL divergence, top-1 agreement and perplexity first. It comes right before this lesson in the course.
Where this lesson picks up: You saw that fewer weight bits cut memory traffic. But each removed bit adds rounding error, so you check quality with perplexity, KL or a task score. That left one question open: which format and which method should you pick for a real deployment?
What comes next
This is the last lesson of the quantization part; next, you compare real candidates on memory, quality and measured speed.
Next lesson: MDPs: discounted returns and the Bellman expectation equation
More lessons in this part
- Scale, zero point and granularity 22 min
- Quantization: rounding error, signal-to-noise and faster decoding 14 min
- Did quantizing hurt the model? KL divergence, top-1 agreement and perplexity 16 min
Take the lesson
Short cards, one idea at a time, with quick checks as you go.