Chapter 4: LLM inference

Prefix caching and cache-aware routing

Many prompts start with the same text, such as a chatbot's fixed instructions. A server that reuses that work gives the first token much sooner, and interviews ask about it. You will be able to compute this saving, say when reuse breaks, and route requests to the right model copy.

25 minInteractive lessonChapter 4: LLM inferenceLesson 9 of 13

What you will learn

  • Compute prefix-cache hit rate and its effect on TTFT
  • Explain block-hash prefix lookup and LRU eviction
  • Design cache-aware routing across replicas

Key terms

prefix
the first tokens of a prompt, which many requests share
hit rate
cached tokens divided by all tokens of a prompt
uncached tail
the tokens after the cached prefix, which prefill must still compute
replica
one copy of the model, with its own KV cache

Prerequisites

Take Continuous batching and paged KV memory first. It comes right before this lesson in the course.

Where this lesson picks up: You kept each request's KV cache in fixed blocks of 16 tokens, and finished requests gave their blocks back. A new request with the same start still computes those keys and values again: can the server reuse them?

What comes next

A cache hit only shortens prefill and decode still makes one token per pass, so can one pass check several tokens at once?

Next lesson: Speculative decoding: the accept rule and expected speedup

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.