Chapter 3: LLM foundations

Scaled dot-product attention

Attention lets each token look at the other tokens and take what it needs from them. "Implement and explain attention" is the most common coding question in LLM interviews. The usual follow-up is: why divide the scores by a square root?

18 minInteractive lessonChapter 3: LLM foundationsLesson 4 of 12

What you will learn

  • Compute attention outputs by hand for a small example
  • Explain why scores are scaled by sqrt(d_k)
  • Implement scaled dot-product attention with batch dimensions

Key terms

query
what a token looks for, as a list of numbers
key
what a token offers to match
value
what a token gives back
softmax
turns scores into weights that add up to 1

Prerequisites

Take Tokenizer failure modes, special tokens and vocabulary size first. It comes right before this lesson in the course.

Where this lesson picks up: In the tokenizer lesson, you saw text become tokens, and each token got its own vector. But each vector still knows only its own token, so how can it use the words around it?

What comes next

Next, each token may use only the words before it, and you run several attentions side by side.

Next lesson: Causal masks and multi-head attention

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.