CLIP, Contrastive Losses and the Limits of Embedding Retrieval

MediumDeep Learning

Reported in interviews at Snap

Problem statement

How would you build text search over a video catalog with a CLIP-style model? Separate text and visual encoders produce normalized embeddings, and an approximate nearest neighbor index retrieves videos by cosine similarity. We will examine training and then a search failure. Explain the dual encoder, calculate its loss and improve difficult queries.

Show hint

Think of each training batch as a classification problem: what are the classes for one video, and where do its negatives come from?

Relevant roles: ML Engineer, Applied Scientist, Research Scientist and AI Engineer.

Related free problems