Elo Ratings for Model Comparison
Problem statement
Pairwise preference leaderboards rank language models with the Elo system: each human vote is a match between two models, and ratings move toward whichever model wins.
Implement elo_rating_update(ratings, matches, k_factor).
ratings: dict from model name to its current rating.matches: list of(model_a, model_b, result)tuples in order, withresultequal to'a'(model_a wins),'b'(model_b wins) or'draw'.k_factor: the step size $K$.
Process the matches sequentially, each one using the ratings produced by the previous matches. For a match between ratings $R_a$ and $R_b$:
$$E_a = \frac{1}{1 + 10^{(R_b - R_a)/400}}, \qquad E_b = 1 - E_a,$$
with actual scores $S_a, S_b$ of 1 and 0 for a win and loss, or 0.5 each for a draw, and updates $R_a \leftarrow R_a + K (S_a - E_a)$, $R_b \leftarrow R_b + K (S_b - E_b)$.
Return a new dict with every model's final rating as a float (models with no matches keep their rating). Do not modify the input dict.
Example
Input:
elo_rating_update({'model_a': 1500, 'model_b': 1500}, [('model_a', 'model_b', 'a')], 32)
Output:
{'model_a': 1516.0, 'model_b': 1484.0}
Reasoning: Equal ratings give $E_a = E_b = 0.5$. Model A wins, so it gains $32 (1 - 0.5) = 16$ and model B loses 16.
Show hint
What function maps a rating difference to a win probability in this system?
Relevant roles: LLM Engineer, AI Engineer, ML Engineer, Research Scientist and Data Scientist.