Chapter 3: LLM foundations
The training bill: memory per parameter and GPU time
Before a lab pays for a run, someone must check the GPU memory and the time it needs. Interviewers ask you to make these quick estimates on the spot.
What you will learn
- Estimate memory for weights, gradients and optimizer states
- Estimate training wall-clock time from 6ND FLOPs and MFU
Key terms
- FLOP
- one multiply or one add of two numbers
- bf16
- a number format that stores each number in 2 bytes
- fp32
- a number format that stores each number in 4 bytes
- MFU
- the share of the peak speed of a GPU that training really uses
- GiB
- a unit of memory: 1024³ bytes
Prerequisites
Take Scaling laws: how big a model for your compute budget first. It comes right before this lesson in the course.
Where this lesson picks up: Last lesson found how big a model a compute budget buys, with $C = 6ND$ FLOPs. Will that model fit in GPU memory, and how long will training take?
What comes next
Training a 7B model with Adam needs about 78 GiB before activations, so the next lesson splits that work across many GPUs.
Next lesson: Parallelism: data, tensor, pipeline and ZeRO/FSDP
More lessons in this part
- Pretraining data: dedup, quality filters and contamination 16 min
- Scaling laws: how big a model for your compute budget 15 min
- Parallelism: data, tensor, pipeline and ZeRO/FSDP 22 min
Take the lesson
Short cards, one idea at a time, with quick checks as you go.