Chapter 6: LLM applications

RAG and hallucination metrics

A RAG bot first searches for text, then writes an answer from it. When an answer is wrong, you must know which of the two parts to fix, or you spend time on the wrong one. Interviewers ask this question often: "Your bot gave a wrong answer. How do you find out what broke?"

18 minInteractive lessonChapter 6: LLM applicationsLesson 9 of 12

What you will learn

  • Compute recall@k and MRR for retrieval
  • Measure faithfulness as supported claims over total claims
  • Diagnose whether a bad answer is a retrieval or generation failure

Key terms

recall@k
the share of helpful chunks found in the top k chunks
MRR
the mean of 1 divided by the rank of each question's first hit
faithfulness
the share of answer claims that the fetched text backs
claim
one single fact in an answer
relevance
whether the answer addresses the question asked

Prerequisites

Take Golden sets and LLM-as-judge first. It comes right before this lesson in the course.

Where this lesson picks up: You built a golden set of questions and an LLM judge that grades each answer against a rubric. But when a RAG bot fails a question, that grade does not say which part of the bot broke.

What comes next

System A scores 0.72 and system B scores 0.68 on 50 questions: is B really worse?

Next lesson: Statistics of small evals: CIs, pass@k and Elo

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.