Chapter 4: LLM inference

Choosing a method: weight-only, W8A8, FP8, GPTQ, AWQ, SmoothQuant

You will be able to pick weight-only INT4, W8A8 or FP8 from what limits a deployment's speed, and explain how GPTQ and SmoothQuant work. Interviewers ask this to see if you choose with numbers and measurements, not by habit.

21 minInteractive lessonChapter 4: LLM inferenceLesson 13 of 13

What you will learn

  • Explain why activation outliers break INT8 and how SmoothQuant fixes them
  • Explain GPTQ's error compensation
  • Choose weight-only vs W8A8 vs FP8 for a latency- or throughput-bound deployment

Key terms

decode
the step that makes each new token, one at a time
prefill
the step that reads the whole prompt at once
activation
a temporary number passed between layers
W8A8
8-bit weights and 8-bit activations
KV cache
stored keys and values that grow with batch and context

Prerequisites

Take Did quantizing hurt the model? KL divergence, top-1 agreement and perplexity first. It comes right before this lesson in the course.

Where this lesson picks up: You saw that fewer weight bits cut memory traffic. But each removed bit adds rounding error, so you check quality with perplexity, KL or a task score. That left one question open: which format and which method should you pick for a real deployment?

What comes next

This is the last lesson of the quantization part; next, you compare real candidates on memory, quality and measured speed.

Next lesson: MDPs: discounted returns and the Bellman expectation equation

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.