Chapter 3: LLM foundations

The transformer block: shapes, parameters and quadratic cost

Language models stack dozens of copies of one block. Interviewers often ask you to count a block's parameters, to say why long inputs cost so much, and to say where the norm goes. In this lesson you build one block and answer all three questions.

18 minInteractive lessonChapter 3: LLM foundationsLesson 6 of 12

What you will learn

  • Assemble a pre-norm decoder block forward pass
  • Count parameters per layer (about 12 d^2) and attention FLOPs
  • Explain why pre-LN trains more stably than post-LN

Key terms

width d
how many numbers store each token
parameters
the learned numbers in a block, mostly in its matrices
block
one repeated unit: attention, then an MLP, each added back
MLP
a small two-layer network that widens each token, then shrinks it
residual stream
the running x that every sub-block adds its note to

Prerequisites

Take Causal masks and multi-head attention first. It comes right before this lesson in the course.

Where this lesson picks up: In the masks lesson, you built causal multi-head attention: each token mixes in itself and earlier tokens, with several heads side by side. One attention layer is not a model yet, so what goes around it, and how big is a full block?

What comes next

Next, you see what today's models changed in this 2017 block: the norm, the MLP and the way they mark positions.

Next lesson: What changed since 2017: RMSNorm, SwiGLU and RoPE

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.