Chapter 6: LLM applications

Statistics of small evals: CIs, pass@k and Elo

Teams often choose between two models from a test of 100 questions or a few hundred votes. Interviewers give you exactly this case: "B gets 72%, A gets 70% on 100 questions. Ship B?" In this lesson you put a range on a score, count pass@k fairly, and turn wins into ratings.

20 minInteractive lessonChapter 6: LLM applicationsLesson 10 of 12

What you will learn

  • Compute a confidence interval or paired test for an eval difference
  • Compute unbiased pass@k from n samples
  • Fit Elo or Bradley-Terry ratings from pairwise wins

Key terms

standard error (SE)
how much a score moves when you re-run the test
95% interval
a range that holds the true score in 95 of 100 re-runs
pass@k
the chance that at least one of k tries is right
Elo rating
a score for each model, built from its wins and losses

Prerequisites

Take RAG and hallucination metrics first. It comes right before this lesson in the course.

Where this lesson picks up: You split RAG quality into a retrieval score and a generation score, and you measured each one on a small test set. But a score from a small test set changes from run to run, so how much can you trust a small difference between two scores?

What comes next

Next, the model uses tools again and again until it finishes a task, so how do you score an agent that can try many times?

Next lesson: An agent is an LLM in a loop with tools

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.