Chapter 7: ML system design and experiments

Sizing: power, minimum detectable effect and duration

With too few users, a real win looks like noise and you miss it. Interviewers ask this right after the design. They expect the inputs, the formula, a worked number and the runtime.

21 minInteractive lessonChapter 7: ML system design and experimentsLesson 11 of 13

What you will learn

  • Compute sample size per arm for a proportion metric
  • Compute the MDE for a fixed traffic budget
  • Convert sample size into test duration

Key terms

false-alarm rate α (alpha)
the chance of a false win or loss when the change does nothing, usually 5%
power
the chance the test catches a real lift, usually set to 80%
minimum detectable effect (MDE)
the smallest lift the test catches with the power you want
variance σ²
the average squared distance of one user's value from the average
users per group n
the number of users who see each version

Prerequisites

Take A/B design: randomization unit, metrics and guardrails first. It comes right before this lesson in the course.

Where this lesson picks up: You fixed the design of an A/B test: the user is the unit, conversion is the one main metric, and the guardrails come before launch. One question stayed open: how many users does the test need, and for how many days?

What comes next

You plan the test for 15 days, but a manager wants to stop on day 4 because the result looks good: what goes wrong?

Next lesson: Many metrics, many tests: Bonferroni, Holm and BH

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.