Chapter 3: LLM foundations
Reading past the training length: RoPE scaling and YaRN
Model cards list a context length, and interviewers ask how a model reads past it. You will see which RoPE pairs break and how YaRN fixes them.
What you will learn
- Explain why RoPE models break past their training length
- Compare position interpolation with YaRN's per-pair rule
Key terms
- training length
- the longest text the model saw in training
- stretch s
- the new text length divided by the training length
- turns
- how many full circles a pair made in training
- YaRN
- a rule that gives each RoPE pair its own speed change
Prerequisites
Take What changed since 2017: RMSNorm, SwiGLU and RoPE first. It comes right before this lesson in the course.
Where this lesson picks up: RoPE turns each pair of q and k by an angle that grows with position. What happens at a position the model never reached in training?
What comes next
Long texts are now possible, but every past token's keys and values must stay in memory: how do we shrink that?
Next lesson: KV-sharing attention: MQA, GQA and MLA
More lessons in this part
- What changed since 2017: RMSNorm, SwiGLU and RoPE 16 min
- KV-sharing attention: MQA, GQA and MLA 16 min
- Mixture of experts: routing, balance, total vs active params 16 min
Take the lesson
Short cards, one idea at a time, with quick checks as you go.