Chapter 6: LLM applications
RAG pipeline: chunking and indexing
A chatbot that looks up a company's own documents can only use a fact that it stored as a piece it can find. Interviews about these systems start with this half. They ask how you cut the documents, how much the pieces overlap, which labels you keep and how many retrieved tokens fit in the prompt.
What you will learn
- Choose chunk size and overlap for a document type
- Attach metadata for filtering and citation
- Compute context budget: chunks x size x k
Key terms
- chunk
- a piece of a document that the system stores and searches
- overlap
- the text that the end of one chunk shares with the start of the next
- token
- a short piece of text, often part of a word, that the model reads
- metadata
- labels stored with each chunk, such as its source file and position
- context budget
- the most tokens the prompt may hold
Prerequisites
Take ANN indexes: HNSW and IVF-PQ trade-offs first. It comes right before this lesson in the course.
Where this lesson picks up: You saw that an ANN index, such as HNSW or IVF-PQ, finds the nearest stored vectors fast. But it finds only the vectors that we chose to store: which pieces of our documents become those vectors, and how?
What comes next
Our facts change every week and need citations: do we retrieve them, fine-tune them into the model, or paste the whole document into the prompt?
Next lesson: RAG vs fine-tuning vs long context
More lessons in this part
Take the lesson
Short cards, one idea at a time, with quick checks as you go.