Chapter 5: Reinforcement learning and post-training
Thompson sampling: picking by sampling your beliefs
Ads, recommendations and tests use Thompson sampling to share traffic between options. Interviewers ask you to run one step by hand and to say how much traffic a brand-new option gets.
What you will learn
- Run one Thompson sampling step with Beta posteriors
Key terms
- arm
- one option, like one slot machine, that pays 1 (a win) or 0 (a loss)
- pull
- one try of an arm
- win chance
- the arm's true chance of paying 1, which we do not know
- belief curve
- a curve over possible win chances; its area over a range is a belief
- draw
- one random win chance picked from a belief curve
Prerequisites
Take Bandits: regret, epsilon-greedy and UCB first. It comes right before this lesson in the course.
Where this lesson picks up: Last lesson, UCB gave each arm its average plus a bonus that is big when the arm has few pulls. Another way to handle that doubt: keep a belief about each arm's win chance, and pick by sampling it.
What comes next
The model is missing something. Do you change the prompt, add documents, or fine-tune?
Next lesson: Prompt, RAG or fine-tune?
More lessons in this part
- MDPs: discounted returns and the Bellman expectation equation 15 min
- Value iteration and the greedy policy 18 min
- Learning from experience: TD error, Q-learning and SARSA 18 min
- Bandits: regret, epsilon-greedy and UCB 16 min
Take the lesson
Short cards, one idea at a time, with quick checks as you go.