Chapter 2: Deep learning
Contrastive learning: word2vec, InfoNCE and CLIP
You will be able to write the InfoNCE loss with in-batch negatives and explain temperature. You will see how word2vec avoids a softmax over a million words. The pairs come from the data, so you need no labels; labelled versions let a label pick the true pairs. Search and photo–caption models use these vectors, and interviewers ask for this loss.
What you will learn
- Write the InfoNCE loss with in-batch negatives
- Explain skip-gram with negative sampling
- Explain how temperature and negative choice cause collapse or slow training
Key terms
- skip-gram
- a center word predicts each context word near it in a sentence
- center word
- the word in the middle, such as purrs
- context word
- a word near the center word, such as cat
- negative sampling
- score the true word and a few random wrong words as yes-or-no questions
Prerequisites
Take RNNs, LSTMs and why gradients vanish first. It comes right before this lesson in the course.
Where this lesson picks up: You saw an LSTM read a sentence as a list of word vectors, with an added cell path that keeps gradients alive. That left one question open: where do good word vectors come from?
What comes next
We now have good word vectors, so why should a model still read them one at a time?
Next lesson: Why transformers replaced RNNs: parallelism and teacher forcing
More lessons in this part
- RNNs, LSTMs and why gradients vanish 7 min
- Why transformers replaced RNNs: parallelism and teacher forcing 22 min
Take the lesson
Short cards, one idea at a time, with quick checks as you go.