Chapter 4: LLM inference
Continuous batching and paged KV memory
A server can often raise its throughput a lot without new GPUs. Two ideas do it. First, refill a free slot at once (continuous batching). Second, store the KV cache in small fixed blocks so more requests fit (paging). Interviewers ask you to step the scheduler and compute the memory waste by hand.
What you will learn
- Simulate continuous versus static batching throughput
- Compute internal fragmentation for a paged KV cache
- Explain chunked prefill and prefill/decode disaggregation
Key terms
- decode slot
- a place in the batch where one request makes 1 token per step
- static batching
- new requests start only when the whole batch is done
- padding
- idle slot work while a batch waits for its longest request
- continuous batching
- a free slot takes the next waiting request at once
- time step
- one round in which every busy slot makes 1 token
Prerequisites
Take Latency metrics and GPU sizing: TTFT, ITL, throughput, goodput first. It comes right before this lesson in the course.
Where this lesson picks up: You measured TTFT, latency, throughput and goodput, and saw that a bigger batch fills more decode slots. But a short answer can finish early and leave its slot empty while a long answer runs.
What comes next
Many requests start with the same long system prompt. Must each request compute its keys and values again?
Next lesson: Prefix caching and cache-aware routing
More lessons in this part
- Prefix caching and cache-aware routing 25 min
- Speculative decoding: the accept rule and expected speedup 18 min
Take the lesson
Short cards, one idea at a time, with quick checks as you go.