Chapter 7: ML system design and experiments

Ads click prediction: pCTR, calibration and expected value

Each ad you see won a fast contest: the system guessed your chance of a click and what the click pays. A guess of the wrong size shows the wrong ad and loses money. Interviewers ask this design often, and they check that you rank by expected value, undo dropped training rows, and handle millions of IDs.

16 minInteractive lessonChapter 7: ML system design and experimentsLesson 6 of 13

What you will learn

  • Rank ads by pCTR times bid and explain why calibration matters
  • Correct pCTR after negative downsampling
  • Explain feature crosses and factorization machines for sparse features

Key terms

pCTR
the predicted chance that one showing gets a click
bid
the money the advertiser pays for one click
negative downsampling
training on only some of the non-click rows
calibration
predicted chances that come true at the stated rate

Prerequisites

Take Search ranking: retrieval, multi-stage ranking and NDCG first. It comes right before this lesson in the course.

Where this lesson picks up: You built a search ranker: a cheap step finds candidates, then heavier rankers sort a few, trained on click labels and judged by NDCG. Search sorts by relevance, but an ad slot must earn money, so what changes when the ranker sorts ads?

What comes next

You can now score one user and one ad well, but which few hundred ads, out of millions, do you even score?

Next lesson: Candidate generation: two-tower and matrix factorization

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.