Chapter 6: LLM applications
RAG and hallucination metrics
A RAG bot first searches for text, then writes an answer from it. When an answer is wrong, you must know which of the two parts to fix, or you spend time on the wrong one. Interviewers ask this question often: "Your bot gave a wrong answer. How do you find out what broke?"
What you will learn
- Compute recall@k and MRR for retrieval
- Measure faithfulness as supported claims over total claims
- Diagnose whether a bad answer is a retrieval or generation failure
Key terms
- recall@k
- the share of helpful chunks found in the top k chunks
- MRR
- the mean of 1 divided by the rank of each question's first hit
- faithfulness
- the share of answer claims that the fetched text backs
- claim
- one single fact in an answer
- relevance
- whether the answer addresses the question asked
Prerequisites
Take Golden sets and LLM-as-judge first. It comes right before this lesson in the course.
Where this lesson picks up: You built a golden set of questions and an LLM judge that grades each answer against a rubric. But when a RAG bot fails a question, that grade does not say which part of the bot broke.
What comes next
System A scores 0.72 and system B scores 0.68 on 50 questions: is B really worse?
Next lesson: Statistics of small evals: CIs, pass@k and Elo
More lessons in this part
Take the lesson
Short cards, one idea at a time, with quick checks as you go.