Chapter 4: LLM inference

Latency metrics and GPU sizing: TTFT, ITL, throughput, goodput

An empty chat screen and slow text need different fixes. These measurements help you explain service quality and cost in interviews.

20 minInteractive lessonChapter 4: LLM inferenceLesson 6 of 12

What you will learn

  • Compute TTFT, ITL and tokens per second from a token timeline
  • Explain the batch-size trade-off between latency and throughput
  • Estimate the minimum GPU count for a model and traffic

Key terms

TTFT
A measure of the wait before text starts.
ITL
A measure of the gaps as text arrives.
TPS
A rate that summarizes one request.

Prerequisites

Take The KV cache and its memory first. It comes right before this lesson in the course.

Where this lesson picks up: You measured the memory for cached keys and values. Once the cache fits, how fast does text appear?

What comes next

When a short answer ends, can another request use its empty batch slot at once?

Next lesson: Continuous batching and paged KV memory

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.