Chapter 7: ML system design and experiments

Fraud and abuse: adversarial, imbalanced, delayed labels

You will be able to lay out the five layers of a payment fraud system and split its past data by time. Fraud is rare, its true answer arrives late, and attackers change their tricks. A plain classifier fails on all three, so "Design a payment fraud system" is a common interview question.

7 minInteractive lessonChapter 7: ML system design and experimentsLesson 3 of 13

What you will learn

  • Frame a fraud problem with label delay and feedback in mind
  • Design features such as velocity and graph signals
  • Lay out a rules + model + review architecture

Key terms

fraud rate
frauds divided by all payments: 8,195 รท 919,767 = 0.891%
alert
a payment that the system flags for a person to check
investigator
a bank worker who checks an alert and marks the payment genuine or fraud
top-100 precision
the share of the 100 riskiest flagged payments that are truly fraud

Prerequisites

Take Back-of-envelope: QPS, storage and latency budgets first. It comes right before this lesson in the course.

Where this lesson picks up: You put numbers on a design: requests per second, storage, and a latency budget in milliseconds per step. That told you how fast each part must be. It did not say what to do when the thing you predict is rare, late and changes on purpose.

What comes next

The model gives a payment a score of 0.7, and investigators can check only a few alerts a day: where do you draw the line?

Next lesson: Cost-weighted thresholds and review capacity

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.