Chapter 7: ML system design and experiments
Many metrics, many tests: Bonferroni, Holm and BH
A launch decision should survive the whole search for a winner. Different corrections protect different kinds of decisions.
What you will learn
- Explain why testing 20 metrics at 0.05 makes a false win likely
- Apply Bonferroni and Holm to a small list of p-values
- Apply Benjamini-Hochberg and explain false discovery rate
- Choose FWER versus FDR for a launch readout
Key terms
- p-value
- chance, under no effect, of a result at least as extreme as the one observed
- discovery
- a test whose no-effect claim you reject; a candidate finding, not proof
Prerequisites
Take Sizing: power, minimum detectable effect and duration first. It comes right before this lesson in the course.
Where this lesson picks up: You can plan the users and duration for one test. Now a dashboard offers twenty chances to call a win.
What comes next
Next, check assignment, repeated looks and users affecting one another.
Next lesson: Pitfalls: SRM, peeking, interference and multiple metrics
More lessons in this part
- A/B design: randomization unit, metrics and guardrails 20 min
- Sizing: power, minimum detectable effect and duration 21 min
- Pitfalls: SRM, peeking, interference and multiple metrics 19 min
- Variance reduction with CUPED 20 min
Take the lesson
Short cards, one idea at a time, with quick checks as you go.