Chapter 3: LLM foundations

Reading past the training length: RoPE scaling and YaRN

Model cards list a context length, and interviewers ask how a model reads past it. You will see which RoPE pairs break and how YaRN fixes them.

15 minInteractive lessonChapter 3: LLM foundationsLesson 8 of 13

What you will learn

  • Explain why RoPE models break past their training length
  • Compare position interpolation with YaRN's per-pair rule

Key terms

training length
the longest text the model saw in training
stretch s
the new text length divided by the training length
turns
how many full circles a pair made in training
YaRN
a rule that gives each RoPE pair its own speed change

Prerequisites

Take What changed since 2017: RMSNorm, SwiGLU and RoPE first. It comes right before this lesson in the course.

Where this lesson picks up: RoPE turns each pair of q and k by an angle that grows with position. What happens at a position the model never reached in training?

What comes next

Long texts are now possible, but every past token's keys and values must stay in memory: how do we shrink that?

Next lesson: KV-sharing attention: MQA, GQA and MLA

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.