Chapter 3: LLM foundations

The training bill: memory per parameter and GPU time

Before a lab pays for a run, someone must check the GPU memory and the time it needs. Interviewers ask you to make these quick estimates on the spot.

12 minInteractive lessonChapter 3: LLM foundationsLesson 13 of 14

What you will learn

  • Estimate memory for weights, gradients and optimizer states
  • Estimate training wall-clock time from 6ND FLOPs and MFU

Key terms

FLOP
one multiply or one add of two numbers
bf16
a number format that stores each number in 2 bytes
fp32
a number format that stores each number in 4 bytes
MFU
the share of the peak speed of a GPU that training really uses
GiB
a unit of memory: 1024³ bytes

Prerequisites

Take Scaling laws: how big a model for your compute budget first. It comes right before this lesson in the course.

Where this lesson picks up: Last lesson found how big a model a compute budget buys, with $C = 6ND$ FLOPs. Will that model fit in GPU memory, and how long will training take?

What comes next

Training a 7B model with Adam needs about 78 GiB before activations, so the next lesson splits that work across many GPUs.

Next lesson: Parallelism: data, tensor, pipeline and ZeRO/FSDP

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.