Evaluate a Message Classifier: Fit, F1 and Uncertainty
Problem statement
How would you evaluate a classifier that sends potentially abusive messages for human review? Abuse is rare. A tree model fits training data much better than validation data, while a linear model performs similarly and less well on both. Test errors cluster by sender account.
Diagnose bias and variance, compute F1, and assess uncertainty.
Show hint
For each model, does it fit its own training data well, and how much worse does it do on data it has not seen?
Relevant roles: ML Engineer, Data Scientist and Applied Scientist.