Chapter 7: ML system design and experiments
Sizing: power, minimum detectable effect and duration
With too few users, a real win looks like noise and you miss it. Interviewers ask this right after the design. They expect the inputs, the formula, a worked number and the runtime.
What you will learn
- Compute sample size per arm for a proportion metric
- Compute the MDE for a fixed traffic budget
- Convert sample size into test duration
Key terms
- false-alarm rate α (alpha)
- the chance of a false win or loss when the change does nothing, usually 5%
- power
- the chance the test catches a real lift, usually set to 80%
- minimum detectable effect (MDE)
- the smallest lift the test catches with the power you want
- variance σ²
- the average squared distance of one user's value from the average
- users per group n
- the number of users who see each version
Prerequisites
Take A/B design: randomization unit, metrics and guardrails first. It comes right before this lesson in the course.
Where this lesson picks up: You fixed the design of an A/B test: the user is the unit, conversion is the one main metric, and the guardrails come before launch. One question stayed open: how many users does the test need, and for how many days?
What comes next
You plan the test for 15 days, but a manager wants to stop on day 4 because the result looks good: what goes wrong?
Next lesson: Many metrics, many tests: Bonferroni, Holm and BH
More lessons in this part
- A/B design: randomization unit, metrics and guardrails 20 min
- Many metrics, many tests: Bonferroni, Holm and BH 16 min
- Pitfalls: SRM, peeking, interference and multiple metrics 19 min
- Variance reduction with CUPED 20 min
Take the lesson
Short cards, one idea at a time, with quick checks as you go.