Chapter 4: LLM inference
Prefix caching and cache-aware routing
Many prompts start with the same text, such as a chatbot's fixed instructions. A server that reuses that work gives the first token much sooner, and interviews ask about it. You will be able to compute this saving, say when reuse breaks, and route requests to the right model copy.
What you will learn
- Compute prefix-cache hit rate and its effect on TTFT
- Explain block-hash prefix lookup and LRU eviction
- Design cache-aware routing across replicas
Key terms
- prefix
- the first tokens of a prompt, which many requests share
- hit rate
- cached tokens divided by all tokens of a prompt
- uncached tail
- the tokens after the cached prefix, which prefill must still compute
- replica
- one copy of the model, with its own KV cache
Prerequisites
Take Continuous batching and paged KV memory first. It comes right before this lesson in the course.
Where this lesson picks up: You kept each request's KV cache in fixed blocks of 16 tokens, and finished requests gave their blocks back. A new request with the same start still computes those keys and values again: can the server reuse them?
What comes next
A cache hit only shortens prefill and decode still makes one token per pass, so can one pass check several tokens at once?
Next lesson: Speculative decoding: the accept rule and expected speedup
More lessons in this part
- Continuous batching and paged KV memory 18 min
- Speculative decoding: the accept rule and expected speedup 18 min
Take the lesson
Short cards, one idea at a time, with quick checks as you go.