Chapter 5: Reinforcement learning and post-training

LoRA and QLoRA: low-rank updates and memory math

Training every weight needs much more GPU memory than the model itself. LoRA freezes the big weights and trains a tiny add-on instead. You will count LoRA's trainable numbers and write its layer and merge in code. You will also estimate the GPU memory of LoRA and QLoRA; interviews ask for all of this.

23 minInteractive lessonChapter 5: Reinforcement learning and post-trainingLesson 7 of 13

What you will learn

  • Implement a LoRA linear layer and merge it into the base weight
  • Compute LoRA trainable parameters for a given rank
  • Estimate memory savings of LoRA and QLoRA vs full fine-tuning

Key terms

LoRA
a way to adapt a model by training a small add-on, not all weights
frozen weight
a weight that training never changes
adapter
the two thin matrices B and A that LoRA trains
rank $r$
the inner width of B and A, which you choose
merge
adding the trained add-on into W to get one matrix

Prerequisites

Take SFT: chat templates and masked loss first. It comes right before this lesson in the course.

Where this lesson picks up: You trained a model with SFT: next-token loss on formatted chats, with the prompt masked out, and every weight changed. That leaves a question: what does it cost to train every weight of a 7B model?

What comes next

Next, you will see how people's comparisons between two answers train a reward model.

Next lesson: From demonstrations to preferences: reward models

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.