Chapter 4: LLM inference

Scale, zero point and granularity

A model can store short integer labels instead of full floating-point weights. In interviews, explain the ruler that gives those labels meaning.

22 minInteractive lessonChapter 4: LLM inferenceLesson 11 of 13

What you will learn

  • Compute symmetric and asymmetric scale and zero point
  • Quantize and dequantize per channel
  • Explain why per-group quantization helps at 4 bits

Key terms

Quantization
Replace model numbers with stored integer codes.
Integer code
A whole-number label that needs a stored ruler.
Reconstruction
Convert stored labels back to approximate numbers.

Prerequisites

Take Speculative decoding: the accept rule and expected speedup first. It comes right before this lesson in the course.

Where this lesson picks up: You reduced token work with a draft model and a check. Now you reduce the storage for model numbers; no draft-model maths is needed.

What comes next

Measure reconstruction error, memory savings and changes in model outputs.

Next lesson: Quantization: rounding error, signal-to-noise and faster decoding

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.