Chapter 4: LLM inference
Latency metrics and GPU sizing: TTFT, ITL, throughput, goodput
An empty chat screen and slow text need different fixes. These measurements help you explain service quality and cost in interviews.
What you will learn
- Compute TTFT, ITL and tokens per second from a token timeline
- Explain the batch-size trade-off between latency and throughput
- Estimate the minimum GPU count for a model and traffic
Key terms
- TTFT
- A measure of the wait before text starts.
- ITL
- A measure of the gaps as text arrives.
- TPS
- A rate that summarizes one request.
Prerequisites
Take The KV cache and its memory first. It comes right before this lesson in the course.
Where this lesson picks up: You measured the memory for cached keys and values. Once the cache fits, how fast does text appear?
What comes next
When a short answer ends, can another request use its empty batch slot at once?
Next lesson: Continuous batching and paged KV memory
More lessons in this part
- Prefill vs decode on the roofline 18 min
- FlashAttention: tiled online softmax and memory traffic 24 min
- The KV cache and its memory 18 min
Take the lesson
Short cards, one idea at a time, with quick checks as you go.