Chapter 7: ML system design and experiments
Cost-weighted thresholds and review capacity
"How would you set the threshold?" is a standard question in fraud and risk interviews. A strong answer starts from the price of each kind of mistake. It also uses how many cases the review team can check in a day, not 0.5 or the best F1.
What you will learn
- Pick a threshold that minimizes expected cost
- Compute precision at the top-k cases reviewers can handle
- Explain cost-sensitive training versus post-hoc thresholding
Key terms
- threshold
- the score at or above which a payment is flagged for a reviewer
- false alarm (FP)
- a fine payment that you flag
- missed fraud (FN)
- a fraud payment that you let pass
- review capacity k
- how many cases the review team can check in a day
- precision at k
- the share of fraud among the top k cases
Prerequisites
Take Fraud and abuse: adversarial, imbalanced, delayed labels first. It comes right before this lesson in the course.
Where this lesson picks up: You saw how a fraud system joins rules, a real-time model and human reviewers, because fraud is rare, labels come late and attackers adapt. But the model only gives each payment a score, so where do you draw the line that sends a payment to the reviewers?
What comes next
The next lesson asks how search finds the best ten results among millions of documents, and how to score a ranked list.
Next lesson: Search ranking: retrieval, multi-stage ranking and NDCG
More lessons in this part
Take the lesson
Short cards, one idea at a time, with quick checks as you go.