Chapter 3: LLM foundations
KV-sharing attention: MQA, GQA and MLA
A chat model keeps the keys and values of all past tokens in memory. For long chats and many users, this memory gets very big. You will count it in bytes and learn three ways to make it smaller. Interviewers often ask why heads share keys and values, and how much memory a model needs.
What you will learn
- Compute KV bytes per token for MHA, GQA, MQA and MLA
- Implement grouped-query attention
- Explain the quality versus memory trade-off of KV sharing
Key terms
- KV cache
- the stored keys and values of all past tokens
- d
- how many numbers one key or one value holds
- byte
- a small unit of computer memory
- MiB
- 1,024 × 1,024 bytes of memory
- GiB
- 1,024 MiB, about 1.07 billion bytes
Prerequisites
Take Reading past the training length: RoPE scaling and YaRN first. It comes right before this lesson in the course.
Where this lesson picks up: You saw how modern models swap in RMSNorm, SwiGLU and RoPE, and RoPE helps them work with long texts. But a long text means the model must keep the key and the value of every past token in memory.
What comes next
The next lesson asks how a model can have a very big MLP but use only a part of it for each token.
Next lesson: Mixture of experts: routing, balance, total vs active params
More lessons in this part
- What changed since 2017: RMSNorm, SwiGLU and RoPE 16 min
- Reading past the training length: RoPE scaling and YaRN 15 min
- Mixture of experts: routing, balance, total vs active params 16 min
Take the lesson
Short cards, one idea at a time, with quick checks as you go.