Chapter 7: ML system design and experiments
Back-of-envelope: QPS, storage and latency budgets
You will turn daily users into QPS, the requests per second, and size the server fleet and its storage. You will also explain why a page that calls many services is slow more often than any one of them. Design interviews often ask for these numbers on the serving box.
What you will learn
- Convert daily users into peak QPS
- Estimate storage for features and embeddings
- Apply Little's law and explain why fan-out inflates tail latency
Key terms
- throughput $\lambda$
- The rate at which requests leave, for example 2 per minute.
- latency $W$
- The time one request spends inside, for example 10 minutes.
- requests inside $L$
- The number of requests inside at once, for example 20.
- Little's law
- $L = \lambda \times W$, true in any steady queue (one that does not grow without end).
Prerequisites
Take The ML system design skeleton first. It comes right before this lesson in the course.
Where this lesson picks up: You learned the design skeleton: clarify, metrics, data, model, serving, rollout. In the serving step you drew a box called model server. How many do you need, and how fast must each one be?
What comes next
Next: when a shop must score each payment before checkout ends, how big a model can you afford?
Next lesson: Fraud and abuse: adversarial, imbalanced, delayed labels
More lessons in this part
Take the lesson
Short cards, one idea at a time, with quick checks as you go.