Macro, Micro and Weighted F1 for a Ticket Router
Problem statement
A support-ticket router assigns each incoming ticket to one of three queues: Billing, Technical and Account closure. Closure tickets are rare, but each one is a customer about to leave, so the support lead wants every queue handled well, not just the big one.
On a held-out set of 1,000 recent tickets the confusion matrix is (rows = true queue, columns = predicted queue):
| True \ Predicted | Billing | Technical | Closure | Total |
|---|---|---|---|---|
| Billing | 760 | 30 | 10 | 800 |
| Technical | 25 | 115 | 10 | 150 |
| Closure | 15 | 15 | 20 | 50 |
You need to decide how to evaluate this multi-class model: which numbers to compute, what each one rewards, and which one should drive model selection.
- Compute the macro-averaged F1.
- Compute the support-weighted F1.
- What does the micro-averaged F1 equal for this model, and why?
- A second model has higher accuracy and weighted F1 but lower macro F1. Which should the team prefer, and on which metric?
- Which metrics would you report for this model, and when is each the right one?
Show hint
For each queue, which total in the matrix goes into precision and which into recall?
Relevant roles: ML Engineer, Data Scientist and Applied Scientist.
Related free problems
- Macro F1 Score for Multi-Class ClassificationML CodingEasy