Chapter 3: LLM foundations
The transformer block: shapes, parameters and quadratic cost
Language models stack dozens of copies of one block. Interviewers often ask you to count a block's parameters, to say why long inputs cost so much, and to say where the norm goes. In this lesson you build one block and answer all three questions.
What you will learn
- Assemble a pre-norm decoder block forward pass
- Count parameters per layer (about 12 d^2) and attention FLOPs
- Explain why pre-LN trains more stably than post-LN
Key terms
- width d
- how many numbers store each token
- parameters
- the learned numbers in a block, mostly in its matrices
- block
- one repeated unit: attention, then an MLP, each added back
- MLP
- a small two-layer network that widens each token, then shrinks it
- residual stream
- the running x that every sub-block adds its note to
Prerequisites
Take Causal masks and multi-head attention first. It comes right before this lesson in the course.
Where this lesson picks up: In the masks lesson, you built causal multi-head attention: each token mixes in itself and earlier tokens, with several heads side by side. One attention layer is not a model yet, so what goes around it, and how big is a full block?
What comes next
Next, you see what today's models changed in this 2017 block: the norm, the MLP and the way they mark positions.
Next lesson: What changed since 2017: RMSNorm, SwiGLU and RoPE
More lessons in this part
Take the lesson
Short cards, one idea at a time, with quick checks as you go.