Chapter 5: Reinforcement learning and post-training
LoRA and QLoRA: low-rank updates and memory math
Training every weight needs much more GPU memory than the model itself. LoRA freezes the big weights and trains a tiny add-on instead. You will count LoRA's trainable numbers and write its layer and merge in code. You will also estimate the GPU memory of LoRA and QLoRA; interviews ask for all of this.
What you will learn
- Implement a LoRA linear layer and merge it into the base weight
- Compute LoRA trainable parameters for a given rank
- Estimate memory savings of LoRA and QLoRA vs full fine-tuning
Key terms
- LoRA
- a way to adapt a model by training a small add-on, not all weights
- frozen weight
- a weight that training never changes
- adapter
- the two thin matrices B and A that LoRA trains
- rank $r$
- the inner width of B and A, which you choose
- merge
- adding the trained add-on into W to get one matrix
Prerequisites
Take SFT: chat templates and masked loss first. It comes right before this lesson in the course.
Where this lesson picks up: You trained a model with SFT: next-token loss on formatted chats, with the prompt masked out, and every weight changed. That leaves a question: what does it cost to train every weight of a 7B model?
What comes next
Next, you will see how people's comparisons between two answers train a reward model.
Next lesson: From demonstrations to preferences: reward models
More lessons in this part
Take the lesson
Short cards, one idea at a time, with quick checks as you go.