Evaluate a Message Classifier: Fit, F1 and Uncertainty

EasyClassical ML

Reported in interviews at Microsoft

Problem statement

How would you evaluate a classifier that sends potentially abusive messages for human review? Abuse is rare. A tree model fits training data much better than validation data, while a linear model performs similarly and less well on both. Test errors cluster by sender account.

Diagnose bias and variance, compute F1, and assess uncertainty.

Show hint

For each model, does it fit its own training data well, and how much worse does it do on data it has not seen?

Relevant roles: ML Engineer, Data Scientist and Applied Scientist.

Related free problems