Annotation Agreement and LLM Judges

MediumLLMs & GenAI

Reported in interviews at Apple

Problem statement

How would you check whether human labels and LLM judgments are reliable enough to guide training or launches? Agreement can look strong when most items get the same label. You need to distinguish real consistency from chance and detect systematic judging biases.

Calculate chance-adjusted agreement, interpret it and address judge bias.

Show hint

How much agreement would two raters reach if each labeled at random with their own pass rate?

Relevant roles: ML Engineer, AI Engineer, Data Scientist and Applied Scientist.

Related free problems