Chapter 7: ML system design and experiments

Many metrics, many tests: Bonferroni, Holm and BH

A launch decision should survive the whole search for a winner. Different corrections protect different kinds of decisions.

16 minInteractive lessonChapter 7: ML system design and experimentsAfter sizing, before pitfalls

What you will learn

  • Explain why testing 20 metrics at 0.05 makes a false win likely
  • Apply Bonferroni and Holm to a small list of p-values
  • Apply Benjamini-Hochberg and explain false discovery rate
  • Choose FWER versus FDR for a launch readout

Key terms

p-value
chance, under no effect, of a result at least as extreme as the one observed
discovery
a test whose no-effect claim you reject; a candidate finding, not proof

Prerequisites

Take Sizing: power, minimum detectable effect and duration first. It comes right before this lesson in the course.

Where this lesson picks up: You can plan the users and duration for one test. Now a dashboard offers twenty chances to call a win.

What comes next

Next, check assignment, repeated looks and users affecting one another.

Next lesson: Pitfalls: SRM, peeking, interference and multiple metrics

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.