Chapter 5: Reinforcement learning and post-training

SFT: chat templates and masked loss

You will be able to turn a chat into one token list and mark, by hand, which tokens the model learns from. A wrong template or a wrong mask breaks a fine-tuned model without an error: it does not stop, or it ignores its instructions. Interviews for fine-tuning work ask exactly how this works.

19 minInteractive lessonChapter 5: Reinforcement learning and post-trainingLesson 6 of 13

What you will learn

  • Format a conversation with a chat template
  • Mask instruction tokens in the loss
  • Explain how SFT data quality and diversity affect the result

Key terms

supervised fine-tuning (SFT)
further training of an existing model on example inputs and the desired replies.
chat template
a fixed way to write a chat as one token list, with role headers.
system message
an instruction about how the model must behave, such as 'Be brief.'.
begin-of-text (ID 1)
the special token that starts every list.

Prerequisites

Take Prompt, RAG or fine-tune? first. It comes right before this lesson in the course.

Where this lesson picks up: In the last lesson you learned to try a prompt first and to fine-tune when prompts cannot get the format or style you need. That left a question: how does a chat become data that a model can train on?

What comes next

Full SFT updates every weight of the model, so could we learn the same change and train far fewer numbers?

Next lesson: LoRA and QLoRA: low-rank updates and memory math

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.