Chapter 4: LLM inference

Continuous batching and paged KV memory

A server can often raise its throughput a lot without new GPUs. Two ideas do it. First, refill a free slot at once (continuous batching). Second, store the KV cache in small fixed blocks so more requests fit (paging). Interviewers ask you to step the scheduler and compute the memory waste by hand.

18 minInteractive lessonChapter 4: LLM inferenceLesson 7 of 12

What you will learn

  • Simulate continuous versus static batching throughput
  • Compute internal fragmentation for a paged KV cache
  • Explain chunked prefill and prefill/decode disaggregation

Key terms

decode slot
a place in the batch where one request makes 1 token per step
static batching
new requests start only when the whole batch is done
padding
idle slot work while a batch waits for its longest request
continuous batching
a free slot takes the next waiting request at once
time step
one round in which every busy slot makes 1 token

Prerequisites

Take Latency metrics and GPU sizing: TTFT, ITL, throughput, goodput first. It comes right before this lesson in the course.

Where this lesson picks up: You measured TTFT, latency, throughput and goodput, and saw that a bigger batch fills more decode slots. But a short answer can finish early and leave its slot empty while a long answer runs.

What comes next

Many requests start with the same long system prompt. Must each request compute its keys and values again?

Next lesson: Prefix caching and cache-aware routing

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.