Chapter 7: ML system design and experiments
Pitfalls: SRM, peeking, interference and multiple metrics
Interviewers hand you a suspicious A/B readout: an odd split, a team that looked daily, a social feature, or one winner among 20 metrics. You must find what broke before you trust any p-value. This lesson gives one check and one fix for each trap.
What you will learn
- Detect sample-ratio mismatch with a chi-square test
- Explain why peeking inflates false positives
- Choose cluster or switchback designs under interference
Key terms
- sample ratio mismatch (SRM)
- arm counts further from the plan than chance allows
- chi-square
- a score that adds up how far each count is from its expected count
- interference
- one user's arm changes another user's outcome
- design effect $\text{Deff}$
- how many times more users a group design needs, because group members act alike
- switchback
- the whole market switches between arms by time block, such as each hour
Prerequisites
Take Many metrics, many tests: Bonferroni, Holm and BH first. It comes right before this lesson in the course.
Where this lesson picks up: You planned a test from the effect, the variance, alpha and power: about 50,000 users per arm and a fixed number of days. But a well-planned test can still give a readout you must not trust. How do you tell?
What comes next
The readout is clean, but the confidence interval is still too wide: can you shrink the noise without more users?
Next lesson: Variance reduction with CUPED
More lessons in this part
- A/B design: randomization unit, metrics and guardrails 20 min
- Sizing: power, minimum detectable effect and duration 21 min
- Many metrics, many tests: Bonferroni, Holm and BH 16 min
- Variance reduction with CUPED 20 min
Take the lesson
Short cards, one idea at a time, with quick checks as you go.