Chapter 3: LLM foundations

Pretraining data: dedup, quality filters and contamination

A web crawl is messy. It holds near-copies of the same page, text of low value, and even test questions. Here you clean it with three tools, in the order an interviewer expects you to describe them.

16 minInteractive lessonChapter 3: LLM foundationsLesson 10 of 12

What you will learn

  • Detect near-duplicate documents with MinHash
  • Explain a classifier-based quality filter and its bias
  • Check a benchmark for n-gram contamination

Key terms

web crawl
a big copy of public web pages, used as training text
benchmark
a fixed test set that scores models
shingle
a short run of words cut from a text, here 2 words
Jaccard similarity J
shared shingles divided by all different shingles of two texts
exact hash
one number for a whole text; one changed word almost always changes it

Prerequisites

Take Mixture of experts: routing, balance, total vs active params first. It comes right before this lesson in the course.

Where this lesson picks up: In the mixture-of-experts lesson, each token went to only a few of many experts, so total parameters grew while the compute per token stayed flat. But more parameters only help if the training text is worth learning, so which text should get in?

What comes next

With a clean corpus, how many tokens and parameters should one compute budget buy?

Next lesson: Scaling laws: how big a model for your compute budget

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.