Chapter 3: LLM foundations

Causal masks and multi-head attention

A language model learns when it guesses the next token. If it can see that token, it learns nothing it can use. Interviewers often ask you to write "multi-head causal self-attention". They check where you block the future, and how you split the work into heads and join it back.

18 minInteractive lessonChapter 3: LLM foundationsLesson 5 of 12

What you will learn

  • Build causal and padding masks and apply them before softmax
  • Split and merge heads with correct reshapes
  • Implement batched multi-head causal self-attention

Key terms

causal mask
a table that says which earlier tokens each token may look at
[PAD]
a filler token that makes short sentences as long as the rest
padding mask
one mark per token: True for a real token, False for [PAD]
head
one of several small attentions that run side by side
Wq, Wk, Wv
learned tables that turn a token into its query, key and value

Prerequisites

Take Scaled dot-product attention first. It comes right before this lesson in the course.

Where this lesson picks up: In the attention lesson, each token scored every other token, turned the scores into weights with softmax and mixed the values. But every token could also look at the tokens after it.

What comes next

Next, you put this attention layer into a full transformer block and count how many numbers the block holds.

Next lesson: The transformer block: shapes, parameters and quadratic cost

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.