Chapter 7: ML system design and experiments

A/B design: randomization unit, metrics and guardrails

Users who chose a feature can differ from users who did not, so a plain gap misleads. An A/B test lets a coin decide who sees the new version. Interviewers often ask how you would A/B test a feature. They expect a design: why a coin flip, which unit, one primary metric, guardrails and a correct standard error.

20 minInteractive lessonChapter 7: ML system design and experimentsLesson 10 of 13

What you will learn

  • Choose the randomization unit and match the analysis unit to it
  • Pick a primary metric and guardrail metrics
  • Handle ratio metrics with the delta method

Key terms

randomization unit
the thing the coin is flipped for, such as a user or a cluster
primary metric
the one number that decides the result of the test
guardrail metric
a number that must not get worse, even if the primary metric wins
ratio metric
one total divided by another, such as clicks per query

Prerequisites

Take BOTEC for LLM products: tokens, GPUs and cost per request first. It comes right before this lesson in the course.

Where this lesson picks up: You priced a new LLM feature per request: tokens, GPUs and the cost of each request. That left one question open: does the feature actually help users?

What comes next

You picked the unit and the metric; next you find how many users and how many days you need to see the effect.

Next lesson: Sizing: power, minimum detectable effect and duration

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.