Chapter 5: Reinforcement learning and post-training

Thompson sampling: picking by sampling your beliefs

Ads, recommendations and tests use Thompson sampling to share traffic between options. Interviewers ask you to run one step by hand and to say how much traffic a brand-new option gets.

14 minInteractive lessonChapter 5: Reinforcement learning and post-trainingLesson 5 of 14

What you will learn

  • Run one Thompson sampling step with Beta posteriors

Key terms

arm
one option, like one slot machine, that pays 1 (a win) or 0 (a loss)
pull
one try of an arm
win chance
the arm's true chance of paying 1, which we do not know
belief curve
a curve over possible win chances; its area over a range is a belief
draw
one random win chance picked from a belief curve

Prerequisites

Take Bandits: regret, epsilon-greedy and UCB first. It comes right before this lesson in the course.

Where this lesson picks up: Last lesson, UCB gave each arm its average plus a bonus that is big when the arm has few pulls. Another way to handle that doubt: keep a belief about each arm's win chance, and pick by sampling it.

What comes next

The model is missing something. Do you change the prompt, add documents, or fine-tune?

Next lesson: Prompt, RAG or fine-tune?

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.