Chapter 4: LLM inference
FlashAttention: tiled online softmax and memory traffic
A score row contains one token's scores against other tokens. You will preserve its softmax result with small persistent state.
What you will learn
- Explain how FlashAttention's tiled online softmax cuts memory traffic
Key terms
- tile
- A small piece of a row of scores.
- frame
- The largest score, which is subtracted from every score before taking the exponential.
Prerequisites
Take Prefill vs decode on the roofline first. It comes right before this lesson in the course.
Where this lesson picks up: You compared memory time with compute time. Can attention avoid a full score row's trip through memory?
What comes next
Examine the memory needed to store past keys and values.
Next lesson: The KV cache and its memory
More lessons in this part
- Prefill vs decode on the roofline 18 min
- The KV cache and its memory 18 min
- Latency metrics and GPU sizing: TTFT, ITL, throughput, goodput 20 min
Take the lesson
Short cards, one idea at a time, with quick checks as you go.