Chapter 3: LLM foundations
Scaling laws: how big a model for your compute budget
Before a lab pays for a big training run, someone must estimate its compute and the best model size for it. Interviewers ask you to make these quick estimates on the spot.
What you will learn
- Estimate training FLOPs with C = 6ND
- Find the compute-optimal model and token count from a budget
Key terms
- FLOP
- one multiply or one add of two numbers
- loss
- a number for how wrong the model is; lower is better
- scaling law
- a formula that predicts the loss from model size and token count
- compute budget
- the total FLOPs a team can spend on one training run
Prerequisites
Take Pretraining data: dedup, quality filters and contamination first. It comes right before this lesson in the course.
Where this lesson picks up: You cleaned a pretraining corpus: you removed duplicates, filtered for quality and removed benchmark leaks. That gave you D clean tokens, but not the model size they should train or the cost of the run.
What comes next
You now know how big a model a budget buys. The next lesson asks whether it fits in GPU memory and how long the run takes.
Next lesson: The training bill: memory per parameter and GPU time
More lessons in this part
- Pretraining data: dedup, quality filters and contamination 16 min
- The training bill: memory per parameter and GPU time 12 min
- Parallelism: data, tensor, pipeline and ZeRO/FSDP 22 min
Take the lesson
Short cards, one idea at a time, with quick checks as you go.