Chapter 6: LLM applications
Statistics of small evals: CIs, pass@k and Elo
Teams often choose between two models from a test of 100 questions or a few hundred votes. Interviewers give you exactly this case: "B gets 72%, A gets 70% on 100 questions. Ship B?" In this lesson you put a range on a score, count pass@k fairly, and turn wins into ratings.
What you will learn
- Compute a confidence interval or paired test for an eval difference
- Compute unbiased pass@k from n samples
- Fit Elo or Bradley-Terry ratings from pairwise wins
Key terms
- standard error (SE)
- how much a score moves when you re-run the test
- 95% interval
- a range that holds the true score in 95 of 100 re-runs
- pass@k
- the chance that at least one of k tries is right
- Elo rating
- a score for each model, built from its wins and losses
Prerequisites
Take RAG and hallucination metrics first. It comes right before this lesson in the course.
Where this lesson picks up: You split RAG quality into a retrieval score and a generation score, and you measured each one on a small test set. But a score from a small test set changes from run to run, so how much can you trust a small difference between two scores?
What comes next
Next, the model uses tools again and again until it finishes a task, so how do you score an agent that can try many times?
Next lesson: An agent is an LLM in a loop with tools
More lessons in this part
Take the lesson
Short cards, one idea at a time, with quick checks as you go.