A Boosted Churn Model With a Huge Train-Test Gap

EasyClassical ML

Reported in interviews at Picnic

Problem statement

A boosted churn model with many depth-50 trees has training AUC 0.95 and test AUC 0.50. Which change most directly reduces its capacity to overfit?

A. Add more trees. B. Use 3-fold instead of 10-fold cross-validation. C. Log-transform skewed features. D. Reduce the maximum tree depth.

Show hint

What does a model that is nearly perfect on training users and random on new users tell you about its capacity?

Relevant roles: ML Engineer, Applied Scientist and Data Scientist.

Related free problems