Chapter 5: Reinforcement learning and post-training
SFT: chat templates and masked loss
You will be able to turn a chat into one token list and mark, by hand, which tokens the model learns from. A wrong template or a wrong mask breaks a fine-tuned model without an error: it does not stop, or it ignores its instructions. Interviews for fine-tuning work ask exactly how this works.
What you will learn
- Format a conversation with a chat template
- Mask instruction tokens in the loss
- Explain how SFT data quality and diversity affect the result
Key terms
- supervised fine-tuning (SFT)
- further training of an existing model on example inputs and the desired replies.
- chat template
- a fixed way to write a chat as one token list, with role headers.
- system message
- an instruction about how the model must behave, such as 'Be brief.'.
- begin-of-text (ID 1)
- the special token that starts every list.
Prerequisites
Take Prompt, RAG or fine-tune? first. It comes right before this lesson in the course.
Where this lesson picks up: In the last lesson you learned to try a prompt first and to fine-tune when prompts cannot get the format or style you need. That left a question: how does a chat become data that a model can train on?
What comes next
Full SFT updates every weight of the model, so could we learn the same change and train far fewer numbers?
Next lesson: LoRA and QLoRA: low-rank updates and memory math
More lessons in this part
Take the lesson
Short cards, one idea at a time, with quick checks as you go.