Chapter 3: LLM foundations
What changed since 2017: RMSNorm, SwiGLU and RoPE
You will be able to name the three swaps in a modern model card, give a reason for each, and write RoPE in code. Interviewers often hand you a model card and ask how it differs from the 2017 transformer.
What you will learn
- Explain why modern decoders use RMSNorm
- Implement a SwiGLU feed-forward block
- Implement rotary position embeddings on query/key pairs
Key terms
- model card
- a short page that lists the parts of a model
- RMSNorm
- a norm that sets the size of a vector
- SwiGLU
- an MLP step where one half of a layer's output controls the other half
- RoPE
- a way to mark position that turns the q and k vectors
Prerequisites
Take The transformer block: shapes, parameters and quadratic cost first. It comes right before this lesson in the course.
Where this lesson picks up: You built the 2017 decoder block: attention plus an MLP, each wrapped in a residual and a norm. Modern models still use that block, so what exactly changed?
What comes next
RoPE turns each pair by an angle that grows with position. What happens at a position the model never reached in training?
Next lesson: Reading past the training length: RoPE scaling and YaRN
More lessons in this part
- Reading past the training length: RoPE scaling and YaRN 15 min
- KV-sharing attention: MQA, GQA and MLA 16 min
- Mixture of experts: routing, balance, total vs active params 16 min
Take the lesson
Short cards, one idea at a time, with quick checks as you go.