Chapter 2: Deep learning

Contrastive learning: word2vec, InfoNCE and CLIP

You will be able to write the InfoNCE loss with in-batch negatives and explain temperature. You will see how word2vec avoids a softmax over a million words. The pairs come from the data, so you need no labels; labelled versions let a label pick the true pairs. Search and photo–caption models use these vectors, and interviewers ask for this loss.

7 minInteractive lessonChapter 2: Deep learningLesson 14 of 20

What you will learn

  • Write the InfoNCE loss with in-batch negatives
  • Explain skip-gram with negative sampling
  • Explain how temperature and negative choice cause collapse or slow training

Key terms

skip-gram
a center word predicts each context word near it in a sentence
center word
the word in the middle, such as purrs
context word
a word near the center word, such as cat
negative sampling
score the true word and a few random wrong words as yes-or-no questions

Prerequisites

Take RNNs, LSTMs and why gradients vanish first. It comes right before this lesson in the course.

Where this lesson picks up: You saw an LSTM read a sentence as a list of word vectors, with an added cell path that keeps gradients alive. That left one question open: where do good word vectors come from?

What comes next

We now have good word vectors, so why should a model still read them one at a time?

Next lesson: Why transformers replaced RNNs: parallelism and teacher forcing

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.