Chapter 4: LLM inference
Speculative decoding: the accept rule and expected speedup
You will be able to run the keep-or-redraw rule of speculative decoding by hand and estimate the speedup. A cheap guesser writes a few tokens. The big model checks them all in one pass, as prefill does. Interviewers ask you to state the rule, show that the output does not change, and estimate the gain.
What you will learn
- Implement the accept/reject rule that preserves the target distribution
- Compute expected accepted tokens from the acceptance rate
- Explain when speculation does not help (high temperature, big batches)
Key terms
- target model
- the big model; the final text must follow its chances
- draft model
- a small, cheap model that guesses the next few tokens
- p and q
- the target's chance (p) and the draft's chance (q) for the same token
- guess
- a token that the draft proposes for the target to check
- pass
- one run of the big model, which reads all its weights from memory
Prerequisites
Take Prefix caching and cache-aware routing first. It comes right before this lesson in the course.
Where this lesson picks up: Requests with the same prompt start reused cached KV blocks, so prefill got cheaper. But decode still runs the big model once per new token, and each pass mostly waits for the weights to arrive from memory.
What comes next
The target's weights are 16-bit numbers: can you store them on a smaller grid of whole numbers?
Next lesson: Scale, zero point and granularity
More lessons in this part
Take the lesson
Short cards, one idea at a time, with quick checks as you go.