Chapter 7: ML system design and experiments
A/B design: randomization unit, metrics and guardrails
Users who chose a feature can differ from users who did not, so a plain gap misleads. An A/B test lets a coin decide who sees the new version. Interviewers often ask how you would A/B test a feature. They expect a design: why a coin flip, which unit, one primary metric, guardrails and a correct standard error.
What you will learn
- Choose the randomization unit and match the analysis unit to it
- Pick a primary metric and guardrail metrics
- Handle ratio metrics with the delta method
Key terms
- randomization unit
- the thing the coin is flipped for, such as a user or a cluster
- primary metric
- the one number that decides the result of the test
- guardrail metric
- a number that must not get worse, even if the primary metric wins
- ratio metric
- one total divided by another, such as clicks per query
Prerequisites
Take BOTEC for LLM products: tokens, GPUs and cost per request first. It comes right before this lesson in the course.
Where this lesson picks up: You priced a new LLM feature per request: tokens, GPUs and the cost of each request. That left one question open: does the feature actually help users?
What comes next
You picked the unit and the metric; next you find how many users and how many days you need to see the effect.
Next lesson: Sizing: power, minimum detectable effect and duration
More lessons in this part
- Sizing: power, minimum detectable effect and duration 21 min
- Many metrics, many tests: Bonferroni, Holm and BH 16 min
- Pitfalls: SRM, peeking, interference and multiple metrics 19 min
- Variance reduction with CUPED 20 min
Take the lesson
Short cards, one idea at a time, with quick checks as you go.