Agreement, Ranking Metrics and Reproducibility in a Model Evaluation Study

EasyClassical ML

Reported in interviews at Thomson Reuters

Problem statement

How would you evaluate summary quality, annotator agreement and retrieval? You are comparing models that summarize support tickets and testing retrieval for a related question-answering feature. The studies include ordered quality ratings with missing responses, paired preferences and ranked retrieval results.

Choose an agreement measure and calculate paired agreement and retrieval quality.

Show hint

For each study, ask what kind of data you have: how many judges, which scale, and whether anything is missing.

Relevant roles: ML Engineer, Data Scientist and Applied Scientist.

Related free problems