Elo Ratings for Model Comparison

MediumLLMs & GenAI

Problem statement

Pairwise preference leaderboards rank language models with the Elo system: each human vote is a match between two models, and ratings move toward whichever model wins.

Implement elo_rating_update(ratings, matches, k_factor).

  • ratings: dict from model name to its current rating.
  • matches: list of (model_a, model_b, result) tuples in order, with result equal to 'a' (model_a wins), 'b' (model_b wins) or 'draw'.
  • k_factor: the step size $K$.

Process the matches sequentially, each one using the ratings produced by the previous matches. For a match between ratings $R_a$ and $R_b$:

$$E_a = \frac{1}{1 + 10^{(R_b - R_a)/400}}, \qquad E_b = 1 - E_a,$$

with actual scores $S_a, S_b$ of 1 and 0 for a win and loss, or 0.5 each for a draw, and updates $R_a \leftarrow R_a + K (S_a - E_a)$, $R_b \leftarrow R_b + K (S_b - E_b)$.

Return a new dict with every model's final rating as a float (models with no matches keep their rating). Do not modify the input dict.

Example

Input:

elo_rating_update({'model_a': 1500, 'model_b': 1500}, [('model_a', 'model_b', 'a')], 32)

Output:

{'model_a': 1516.0, 'model_b': 1484.0}

Reasoning: Equal ratings give $E_a = E_b = 0.5$. Model A wins, so it gains $32 (1 - 0.5) = 16$ and model B loses 16.

Show hint

What function maps a rating difference to a win probability in this system?

Relevant roles: LLM Engineer, AI Engineer, ML Engineer, Research Scientist and Data Scientist.

Related free problems