Chapter 3: LLM foundations

Scaling laws: how big a model for your compute budget

Before a lab pays for a big training run, someone must estimate its compute and the best model size for it. Interviewers ask you to make these quick estimates on the spot.

15 minInteractive lessonChapter 3: LLM foundationsLesson 12 of 14

What you will learn

  • Estimate training FLOPs with C = 6ND
  • Find the compute-optimal model and token count from a budget

Key terms

FLOP
one multiply or one add of two numbers
loss
a number for how wrong the model is; lower is better
scaling law
a formula that predicts the loss from model size and token count
compute budget
the total FLOPs a team can spend on one training run

Prerequisites

Take Pretraining data: dedup, quality filters and contamination first. It comes right before this lesson in the course.

Where this lesson picks up: You cleaned a pretraining corpus: you removed duplicates, filtered for quality and removed benchmark leaks. That gave you D clean tokens, but not the model size they should train or the cost of the run.

What comes next

You now know how big a model a budget buys. The next lesson asks whether it fits in GPU memory and how long the run takes.

Next lesson: The training bill: memory per parameter and GPU time

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.