Chapter 7: ML system design and experiments
BOTEC for LLM products: tokens, GPUs and cost per request
Interviews for LLM products often ask what a feature costs each month, and whether to call an API or run your own GPUs. You will answer with a quick estimate: tokens per request, dollars per million tokens, a break-even volume, then a GPU-count check and a latency check.
What you will learn
- Estimate cost per 1,000 requests from token counts and prices
- Find the break-even point between an API and dedicated GPUs
- Estimate how batch size trades latency for cost per token
Key terms
- dollars per million tokens
- the price unit that compares an API with your own GPU
- break-even volume $N^*$
- the monthly tokens where two monthly bills are equal
- utilization $u$
- the share of hours your GPU is busy
- ITL
- the wait between two output tokens of one answer
Prerequisites
Take Ranking and re-ranking: multi-objective, diversity, cold start first. It comes right before this lesson in the course.
Where this lesson picks up: You built a ranker that scores and re-ranks every feed. Now the product adds an LLM call to each request, and a new question opens: what does each call cost?
What comes next
Your own GPUs can look cheaper on paper, so the next lesson asks how to test a switch before every user gets it.
Next lesson: A/B design: randomization unit, metrics and guardrails
Take the lesson
Short cards, one idea at a time, with quick checks as you go.