Agreement, Ranking Metrics and Reproducibility in a Model Evaluation Study
Problem statement
How would you evaluate summary quality, annotator agreement and retrieval? You are comparing models that summarize support tickets and testing retrieval for a related question-answering feature. The studies include ordered quality ratings with missing responses, paired preferences and ranked retrieval results.
Choose an agreement measure and calculate paired agreement and retrieval quality.
Show hint
For each study, ask what kind of data you have: how many judges, which scale, and whether anything is missing.
Relevant roles: ML Engineer, Data Scientist and Applied Scientist.
Related free problems
- Annotation Agreement and LLM JudgesLLMs & GenAIMedium