Chapter 6: LLM applications

RAG pipeline: chunking and indexing

A chatbot that looks up a company's own documents can only use a fact that it stored as a piece it can find. Interviews about these systems start with this half. They ask how you cut the documents, how much the pieces overlap, which labels you keep and how many retrieved tokens fit in the prompt.

22 minInteractive lessonChapter 6: LLM applicationsLesson 6 of 12

What you will learn

  • Choose chunk size and overlap for a document type
  • Attach metadata for filtering and citation
  • Compute context budget: chunks x size x k

Key terms

chunk
a piece of a document that the system stores and searches
overlap
the text that the end of one chunk shares with the start of the next
token
a short piece of text, often part of a word, that the model reads
metadata
labels stored with each chunk, such as its source file and position
context budget
the most tokens the prompt may hold

Prerequisites

Take ANN indexes: HNSW and IVF-PQ trade-offs first. It comes right before this lesson in the course.

Where this lesson picks up: You saw that an ANN index, such as HNSW or IVF-PQ, finds the nearest stored vectors fast. But it finds only the vectors that we chose to store: which pieces of our documents become those vectors, and how?

What comes next

Our facts change every week and need citations: do we retrieve them, fine-tune them into the model, or paste the whole document into the prompt?

Next lesson: RAG vs fine-tuning vs long context

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.