Chapter 7: ML system design and experiments
Ads click prediction: pCTR, calibration and expected value
Each ad you see won a fast contest: the system guessed your chance of a click and what the click pays. A guess of the wrong size shows the wrong ad and loses money. Interviewers ask this design often, and they check that you rank by expected value, undo dropped training rows, and handle millions of IDs.
What you will learn
- Rank ads by pCTR times bid and explain why calibration matters
- Correct pCTR after negative downsampling
- Explain feature crosses and factorization machines for sparse features
Key terms
- pCTR
- the predicted chance that one showing gets a click
- bid
- the money the advertiser pays for one click
- negative downsampling
- training on only some of the non-click rows
- calibration
- predicted chances that come true at the stated rate
Prerequisites
Take Search ranking: retrieval, multi-stage ranking and NDCG first. It comes right before this lesson in the course.
Where this lesson picks up: You built a search ranker: a cheap step finds candidates, then heavier rankers sort a few, trained on click labels and judged by NDCG. Search sorts by relevance, but an ad slot must earn money, so what changes when the ranker sorts ads?
What comes next
You can now score one user and one ad well, but which few hundred ads, out of millions, do you even score?
Next lesson: Candidate generation: two-tower and matrix factorization
More lessons in this part
Take the lesson
Short cards, one idea at a time, with quick checks as you go.