Chapter 4: LLM inference

FlashAttention: tiled online softmax and memory traffic

A score row contains one token's scores against other tokens. You will preserve its softmax result with small persistent state.

24 minInteractive lessonChapter 4: LLM inferenceLesson 5 of 13

What you will learn

  • Explain how FlashAttention's tiled online softmax cuts memory traffic

Key terms

tile
A small piece of a row of scores.
frame
The largest score, which is subtracted from every score before taking the exponential.

Prerequisites

Take Prefill vs decode on the roofline first. It comes right before this lesson in the course.

Where this lesson picks up: You compared memory time with compute time. Can attention avoid a full score row's trip through memory?

What comes next

Examine the memory needed to store past keys and values.

Next lesson: The KV cache and its memory

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.