Chapter 7: ML system design and experiments

Back-of-envelope: QPS, storage and latency budgets

You will turn daily users into QPS, the requests per second, and size the server fleet and its storage. You will also explain why a page that calls many services is slow more often than any one of them. Design interviews often ask for these numbers on the serving box.

16 minInteractive lessonChapter 7: ML system design and experimentsLesson 2 of 13

What you will learn

  • Convert daily users into peak QPS
  • Estimate storage for features and embeddings
  • Apply Little's law and explain why fan-out inflates tail latency

Key terms

throughput $\lambda$
The rate at which requests leave, for example 2 per minute.
latency $W$
The time one request spends inside, for example 10 minutes.
requests inside $L$
The number of requests inside at once, for example 20.
Little's law
$L = \lambda \times W$, true in any steady queue (one that does not grow without end).

Prerequisites

Take The ML system design skeleton first. It comes right before this lesson in the course.

Where this lesson picks up: You learned the design skeleton: clarify, metrics, data, model, serving, rollout. In the serving step you drew a box called model server. How many do you need, and how fast must each one be?

What comes next

Next: when a shop must score each payment before checkout ends, how big a model can you afford?

Next lesson: Fraud and abuse: adversarial, imbalanced, delayed labels

More lessons in this part

Take the lesson

Short cards, one idea at a time, with quick checks as you go.