Chapter 3: LLM foundations

What changed since 2017: RMSNorm, SwiGLU and RoPE

You will be able to name the three swaps in a modern model card, give a reason for each, and write RoPE in code. Interviewers often hand you a model card and ask how it differs from the 2017 transformer.

16 minInteractive lessonChapter 3: LLM foundationsLesson 7 of 13

What you will learn

  • Explain why modern decoders use RMSNorm
  • Implement a SwiGLU feed-forward block
  • Implement rotary position embeddings on query/key pairs

Key terms

model card
a short page that lists the parts of a model
RMSNorm
a norm that sets the size of a vector
SwiGLU
an MLP step where one half of a layer's output controls the other half
RoPE
a way to mark position that turns the q and k vectors

Prerequisites

Take The transformer block: shapes, parameters and quadratic cost first. It comes right before this lesson in the course.

Where this lesson picks up: You built the 2017 decoder block: attention plus an MLP, each wrapped in a residual and a norm. Modern models still use that block, so what exactly changed?

What comes next

RoPE turns each pair by an angle that grows with position. What happens at a position the model never reached in training?

Next lesson: Reading past the training length: RoPE scaling and YaRN

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.