Chapter 3: LLM foundations

KV-sharing attention: MQA, GQA and MLA

A chat model keeps the keys and values of all past tokens in memory. For long chats and many users, this memory gets very big. You will count it in bytes and learn three ways to make it smaller. Interviewers often ask why heads share keys and values, and how much memory a model needs.

16 minInteractive lessonChapter 3: LLM foundationsLesson 8 of 12

What you will learn

  • Compute KV bytes per token for MHA, GQA, MQA and MLA
  • Implement grouped-query attention
  • Explain the quality versus memory trade-off of KV sharing

Key terms

KV cache
the stored keys and values of all past tokens
d
how many numbers one key or one value holds
byte
a small unit of computer memory
MiB
1,024 × 1,024 bytes of memory
GiB
1,024 MiB, about 1.07 billion bytes

Prerequisites

Take Reading past the training length: RoPE scaling and YaRN first. It comes right before this lesson in the course.

Where this lesson picks up: You saw how modern models swap in RMSNorm, SwiGLU and RoPE, and RoPE helps them work with long texts. But a long text means the model must keep the key and the value of every past token in memory.

What comes next

The next lesson asks how a model can have a very big MLP but use only a part of it for each token.

Next lesson: Mixture of experts: routing, balance, total vs active params

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.