CLIP, Contrastive Losses and the Limits of Embedding Retrieval
Problem statement
How would you build text search over a video catalog with a CLIP-style model? Separate text and visual encoders produce normalized embeddings, and an approximate nearest neighbor index retrieves videos by cosine similarity. We will examine training and then a search failure. Explain the dual encoder, calculate its loss and improve difficult queries.
Show hint
Think of each training batch as a classification problem: what are the classes for one video, and where do its negatives come from?
Relevant roles: ML Engineer, Applied Scientist, Research Scientist and AI Engineer.
Related free problems
- Contrastive Loss (InfoNCE / NT-Xent)ML CodingMedium