Chapter 3: LLM foundations
Pretraining data: dedup, quality filters and contamination
A web crawl is messy. It holds near-copies of the same page, text of low value, and even test questions. Here you clean it with three tools, in the order an interviewer expects you to describe them.
What you will learn
- Detect near-duplicate documents with MinHash
- Explain a classifier-based quality filter and its bias
- Check a benchmark for n-gram contamination
Key terms
- web crawl
- a big copy of public web pages, used as training text
- benchmark
- a fixed test set that scores models
- shingle
- a short run of words cut from a text, here 2 words
- Jaccard similarity J
- shared shingles divided by all different shingles of two texts
- exact hash
- one number for a whole text; one changed word almost always changes it
Prerequisites
Take Mixture of experts: routing, balance, total vs active params first. It comes right before this lesson in the course.
Where this lesson picks up: In the mixture-of-experts lesson, each token went to only a few of many experts, so total parameters grew while the compute per token stayed flat. But more parameters only help if the training text is worth learning, so which text should get in?
What comes next
With a clean corpus, how many tokens and parameters should one compute budget buy?
Next lesson: Scaling laws: how big a model for your compute budget
More lessons in this part
- Scaling laws: how big a model for your compute budget 15 min
- The training bill: memory per parameter and GPU time 12 min
- Parallelism: data, tensor, pipeline and ZeRO/FSDP 22 min
Take the lesson
Short cards, one idea at a time, with quick checks as you go.