Chapter 4: LLM inference

Speculative decoding: the accept rule and expected speedup

You will be able to run the keep-or-redraw rule of speculative decoding by hand and estimate the speedup. A cheap guesser writes a few tokens. The big model checks them all in one pass, as prefill does. Interviewers ask you to state the rule, show that the output does not change, and estimate the gain.

18 minInteractive lessonChapter 4: LLM inferenceLesson 10 of 13

What you will learn

  • Implement the accept/reject rule that preserves the target distribution
  • Compute expected accepted tokens from the acceptance rate
  • Explain when speculation does not help (high temperature, big batches)

Key terms

target model
the big model; the final text must follow its chances
draft model
a small, cheap model that guesses the next few tokens
p and q
the target's chance (p) and the draft's chance (q) for the same token
guess
a token that the draft proposes for the target to check
pass
one run of the big model, which reads all its weights from memory

Prerequisites

Take Prefix caching and cache-aware routing first. It comes right before this lesson in the course.

Where this lesson picks up: Requests with the same prompt start reused cached KV blocks, so prefill got cheaper. But decode still runs the big model once per new token, and each pass mostly waits for the weights to arrive from memory.

What comes next

The target's weights are 16-bit numbers: can you store them on a smaller grid of whole numbers?

Next lesson: Scale, zero point and granularity

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.