Choosing the Offline Metric for Candidate Generation

Reported in interviews at TikTok

Problem statement

How would you evaluate the videos a recommender retrieves before ranking them? Several sources search a huge catalog, then a heavier model chooses what users see. You have held-out records of videos users later finished watching. Calculate retrieval coverage and explain how to choose and evaluate a retriever.

Show hint

Think about what the candidate stage is responsible for and what happens to an item it leaves out.

Relevant roles: ML Engineer, Applied Scientist and Data Scientist.

Related free problems