Chapter 7: ML system design and experiments

Cost-weighted thresholds and review capacity

"How would you set the threshold?" is a standard question in fraud and risk interviews. A strong answer starts from the price of each kind of mistake. It also uses how many cases the review team can check in a day, not 0.5 or the best F1.

18 minInteractive lessonChapter 7: ML system design and experimentsLesson 4 of 13

What you will learn

  • Pick a threshold that minimizes expected cost
  • Compute precision at the top-k cases reviewers can handle
  • Explain cost-sensitive training versus post-hoc thresholding

Key terms

threshold
the score at or above which a payment is flagged for a reviewer
false alarm (FP)
a fine payment that you flag
missed fraud (FN)
a fraud payment that you let pass
review capacity k
how many cases the review team can check in a day
precision at k
the share of fraud among the top k cases

Prerequisites

Take Fraud and abuse: adversarial, imbalanced, delayed labels first. It comes right before this lesson in the course.

Where this lesson picks up: You saw how a fraud system joins rules, a real-time model and human reviewers, because fraud is rare, labels come late and attackers adapt. But the model only gives each payment a score, so where do you draw the line that sends a payment to the reviewers?

What comes next

The next lesson asks how search finds the best ten results among millions of documents, and how to score a ranked list.

Next lesson: Search ranking: retrieval, multi-stage ranking and NDCG

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.