Chapter 3: LLM foundations
Causal masks and multi-head attention
A language model learns when it guesses the next token. If it can see that token, it learns nothing it can use. Interviewers often ask you to write "multi-head causal self-attention". They check where you block the future, and how you split the work into heads and join it back.
What you will learn
- Build causal and padding masks and apply them before softmax
- Split and merge heads with correct reshapes
- Implement batched multi-head causal self-attention
Key terms
- causal mask
- a table that says which earlier tokens each token may look at
- [PAD]
- a filler token that makes short sentences as long as the rest
- padding mask
- one mark per token: True for a real token, False for [PAD]
- head
- one of several small attentions that run side by side
- Wq, Wk, Wv
- learned tables that turn a token into its query, key and value
Prerequisites
Take Scaled dot-product attention first. It comes right before this lesson in the course.
Where this lesson picks up: In the attention lesson, each token scored every other token, turned the scores into weights with softmax and mixed the values. But every token could also look at the tokens after it.
What comes next
Next, you put this attention layer into a full transformer block and count how many numbers the block holds.
Next lesson: The transformer block: shapes, parameters and quadratic cost
More lessons in this part
- Scaled dot-product attention 18 min
- The transformer block: shapes, parameters and quadratic cost 18 min
Take the lesson
Short cards, one idea at a time, with quick checks as you go.