Macro, Micro and Weighted F1 for a Ticket Router

MediumClassical ML

Reported in interviews at Google

Problem statement

A support-ticket router assigns each incoming ticket to one of three queues: Billing, Technical and Account closure. Closure tickets are rare, but each one is a customer about to leave, so the support lead wants every queue handled well, not just the big one.

On a held-out set of 1,000 recent tickets the confusion matrix is (rows = true queue, columns = predicted queue):

True \ PredictedBillingTechnicalClosureTotal
Billing7603010800
Technical2511510150
Closure15152050

You need to decide how to evaluate this multi-class model: which numbers to compute, what each one rewards, and which one should drive model selection.

  1. Compute the macro-averaged F1.
  2. Compute the support-weighted F1.
  3. What does the micro-averaged F1 equal for this model, and why?
  4. A second model has higher accuracy and weighted F1 but lower macro F1. Which should the team prefer, and on which metric?
  5. Which metrics would you report for this model, and when is each the right one?
Show hint

For each queue, which total in the matrix goes into precision and which into recall?

Relevant roles: ML Engineer, Data Scientist and Applied Scientist.

Related free problems