Chapter 3: LLM foundations
Scaled dot-product attention
Attention lets each token look at the other tokens and take what it needs from them. "Implement and explain attention" is the most common coding question in LLM interviews. The usual follow-up is: why divide the scores by a square root?
What you will learn
- Compute attention outputs by hand for a small example
- Explain why scores are scaled by sqrt(d_k)
- Implement scaled dot-product attention with batch dimensions
Key terms
- query
- what a token looks for, as a list of numbers
- key
- what a token offers to match
- value
- what a token gives back
- softmax
- turns scores into weights that add up to 1
Prerequisites
Take Tokenizer failure modes, special tokens and vocabulary size first. It comes right before this lesson in the course.
Where this lesson picks up: In the tokenizer lesson, you saw text become tokens, and each token got its own vector. But each vector still knows only its own token, so how can it use the words around it?
What comes next
Next, each token may use only the words before it, and you run several attentions side by side.
Next lesson: Causal masks and multi-head attention
More lessons in this part
- Causal masks and multi-head attention 18 min
- The transformer block: shapes, parameters and quadratic cost 18 min
Take the lesson
Short cards, one idea at a time, with quick checks as you go.