Chapter 3: LLM foundations
Parallelism: data, tensor, pipeline and ZeRO/FSDP
You will be able to compute how long GPUs take to add their gradients and how much time a pipeline of GPUs sits idle. You will also choose a layout for a model that does not fit one GPU. Interviewers ask this to see if you know what each split costs.
What you will learn
- Compute ring all-reduce volume and time
- Compute the pipeline bubble fraction
- Choose a parallelism layout for a model that does not fit one GPU
Key terms
- GPU
- a chip that does the maths of training and has its own memory
- batch
- the examples that one training step uses together
- data parallel
- every GPU holds the whole model and trains on its own slice of the batch
- gradient
- the numbers that say how to change each weight
Prerequisites
Take The training bill: memory per parameter and GPU time first. It comes right before this lesson in the course.
Where this lesson picks up: Last lesson: training with Adam and an fp32 master copy needs about 16 bytes per parameter, so a 7B model needs 112 GB before activations. One 80 GB GPU cannot hold that. So how do you train it?
What comes next
The model now trains on many GPUs. How do we serve it fast to many users at once?
Next lesson: Greedy, beam, top-k and top-p decoding
More lessons in this part
- Pretraining data: dedup, quality filters and contamination 16 min
- Scaling laws: how big a model for your compute budget 15 min
- The training bill: memory per parameter and GPU time 12 min
Take the lesson
Short cards, one idea at a time, with quick checks as you go.