Shipping a Churn Classifier: Splits, Imbalance and Thresholds

HardClassical ML

Reported in interviews at OpenAI

Problem statement

How would you train a churn classifier that a retention team can use? Monthly subscription snapshots contain user history, missing values and a rare cancellation label. The model scores active users each month so the team can decide who receives a discount offer.

Choose data splits, derive an offer threshold and check probability calibration.

Show hint

For every step that learns something from data (a split, an imputer, a resampler, a calibrator, a threshold), ask which rows it is allowed to see.

Relevant roles: ML Engineer, Data Scientist and Applied Scientist.

Related free problems