Chapter 7: ML system design and experiments
Search ranking: retrieval, multi-stage ranking and NDCG
A shop search must pick about 10 results from billions of products while the user waits. "Design product search" is one of the most common design questions in machine-learning interviews. Interviewers expect a funnel sized by item counts, labels built from what users did, and a ranking score that you can compute by hand.
What you will learn
- Design a retrieval + ranking + re-ranking funnel with latency budgets
- Derive relevance labels from clicks and explain position bias
- Compute NDCG@k
Key terms
- inverted index
- a list that maps each word to the items that contain it
- recall
- the share of all good items that a step keeps
- pairwise ranking
- a model that decides which of two items should sit higher
- position bias
- users click high results more, just because they are high
- NDCG
- a score for a ranked list that counts early good items more
Prerequisites
Take Cost-weighted thresholds and review capacity first. It comes right before this lesson in the course.
Where this lesson picks up: You set a risk system's threshold from what each error costs and how many cases the reviewers can check each day. A budget shaped one number, so can a budget shape a whole design?
What comes next
Search sorts by relevance, but ads must sort by the money a click is worth, so what changes?
Next lesson: Ads click prediction: pCTR, calibration and expected value
More lessons in this part
Take the lesson
Short cards, one idea at a time, with quick checks as you go.