Chapter 3: LLM foundations
Mixture of experts: routing, balance, total vs active params
An MoE layer keeps many copies of the FFN block and runs only a few of them for each token. So the model grows, but each token stays cheap. Interviewers ask you to count its parameters and to explain how its router can fail.
What you will learn
- Implement top-k routing with a load-balancing auxiliary loss
- Compute total versus active parameters for an MoE config
- Explain expert collapse and capacity limits
Key terms
- expert
- one copy of the FFN block inside an MoE layer
- router
- a small learned table that scores every expert for a token
- top-k
- keep only the k experts with the highest chances
- gate
- the weight, from 0 to 1, on one picked expert's output
Prerequisites
Take KV-sharing attention: MQA, GQA and MLA first. It comes right before this lesson in the course.
Where this lesson picks up: Last lesson, you shrank the KV cache by sharing keys and values across heads, because memory sets the cost of long contexts. The other big block of each layer is the FFN: can it hold much more without more work per token?
What comes next
Next: with a huge model that is cheap per token, what data do you train it on, and how do you keep that data clean?
Next lesson: Pretraining data: dedup, quality filters and contamination
More lessons in this part
- What changed since 2017: RMSNorm, SwiGLU and RoPE 16 min
- Reading past the training length: RoPE scaling and YaRN 15 min
- KV-sharing attention: MQA, GQA and MLA 16 min
Take the lesson
Short cards, one idea at a time, with quick checks as you go.