Chapter 3: LLM foundations

Parallelism: data, tensor, pipeline and ZeRO/FSDP

You will be able to compute how long GPUs take to add their gradients and how much time a pipeline of GPUs sits idle. You will also choose a layout for a model that does not fit one GPU. Interviewers ask this to see if you know what each split costs.

22 minInteractive lessonChapter 3: LLM foundationsLesson 13 of 13

What you will learn

  • Compute ring all-reduce volume and time
  • Compute the pipeline bubble fraction
  • Choose a parallelism layout for a model that does not fit one GPU

Key terms

GPU
a chip that does the maths of training and has its own memory
batch
the examples that one training step uses together
data parallel
every GPU holds the whole model and trains on its own slice of the batch
gradient
the numbers that say how to change each weight

Prerequisites

Take The training bill: memory per parameter and GPU time first. It comes right before this lesson in the course.

Where this lesson picks up: Last lesson: training with Adam and an fp32 master copy needs about 16 bytes per parameter, so a 7B model needs 112 GB before activations. One 80 GB GPU cannot hold that. So how do you train it?

What comes next

The model now trains on many GPUs. How do we serve it fast to many users at once?

Next lesson: Greedy, beam, top-k and top-p decoding

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.