Chapter 7: ML system design and experiments

Search ranking: retrieval, multi-stage ranking and NDCG

A shop search must pick about 10 results from billions of products while the user waits. "Design product search" is one of the most common design questions in machine-learning interviews. Interviewers expect a funnel sized by item counts, labels built from what users did, and a ranking score that you can compute by hand.

18 minInteractive lessonChapter 7: ML system design and experimentsLesson 5 of 13

What you will learn

  • Design a retrieval + ranking + re-ranking funnel with latency budgets
  • Derive relevance labels from clicks and explain position bias
  • Compute NDCG@k

Key terms

inverted index
a list that maps each word to the items that contain it
recall
the share of all good items that a step keeps
pairwise ranking
a model that decides which of two items should sit higher
position bias
users click high results more, just because they are high
NDCG
a score for a ranked list that counts early good items more

Prerequisites

Take Cost-weighted thresholds and review capacity first. It comes right before this lesson in the course.

Where this lesson picks up: You set a risk system's threshold from what each error costs and how many cases the reviewers can check each day. A budget shaped one number, so can a budget shape a whole design?

What comes next

Search sorts by relevance, but ads must sort by the money a click is worth, so what changes?

Next lesson: Ads click prediction: pCTR, calibration and expected value

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.