Chapter 3: LLM foundations

Mixture of experts: routing, balance, total vs active params

An MoE layer keeps many copies of the FFN block and runs only a few of them for each token. So the model grows, but each token stays cheap. Interviewers ask you to count its parameters and to explain how its router can fail.

16 minInteractive lessonChapter 3: LLM foundationsLesson 9 of 12

What you will learn

  • Implement top-k routing with a load-balancing auxiliary loss
  • Compute total versus active parameters for an MoE config
  • Explain expert collapse and capacity limits

Key terms

expert
one copy of the FFN block inside an MoE layer
router
a small learned table that scores every expert for a token
top-k
keep only the k experts with the highest chances
gate
the weight, from 0 to 1, on one picked expert's output

Prerequisites

Take KV-sharing attention: MQA, GQA and MLA first. It comes right before this lesson in the course.

Where this lesson picks up: Last lesson, you shrank the KV cache by sharing keys and values across heads, because memory sets the cost of long contexts. The other big block of each layer is the FFN: can it hold much more without more work per token?

What comes next

Next: with a huge model that is cheap per token, what data do you train it on, and how do you keep that data clean?

Next lesson: Pretraining data: dedup, quality filters and contamination

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.