Chapter 7: ML system design and experiments

Pitfalls: SRM, peeking, interference and multiple metrics

Interviewers hand you a suspicious A/B readout: an odd split, a team that looked daily, a social feature, or one winner among 20 metrics. You must find what broke before you trust any p-value. This lesson gives one check and one fix for each trap.

19 minInteractive lessonChapter 7: ML system design and experimentsLesson 12 of 13

What you will learn

  • Detect sample-ratio mismatch with a chi-square test
  • Explain why peeking inflates false positives
  • Choose cluster or switchback designs under interference

Key terms

sample ratio mismatch (SRM)
arm counts further from the plan than chance allows
chi-square
a score that adds up how far each count is from its expected count
interference
one user's arm changes another user's outcome
design effect $\text{Deff}$
how many times more users a group design needs, because group members act alike
switchback
the whole market switches between arms by time block, such as each hour

Prerequisites

Take Many metrics, many tests: Bonferroni, Holm and BH first. It comes right before this lesson in the course.

Where this lesson picks up: You planned a test from the effect, the variance, alpha and power: about 50,000 users per arm and a fixed number of days. But a well-planned test can still give a readout you must not trust. How do you tell?

What comes next

The readout is clean, but the confidence interval is still too wide: can you shrink the noise without more users?

Next lesson: Variance reduction with CUPED

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.