Chapter 7: ML system design and experiments

BOTEC for LLM products: tokens, GPUs and cost per request

Interviews for LLM products often ask what a feature costs each month, and whether to call an API or run your own GPUs. You will answer with a quick estimate: tokens per request, dollars per million tokens, a break-even volume, then a GPU-count check and a latency check.

20 minInteractive lessonChapter 7: ML system design and experimentsLesson 9 of 13

What you will learn

  • Estimate cost per 1,000 requests from token counts and prices
  • Find the break-even point between an API and dedicated GPUs
  • Estimate how batch size trades latency for cost per token

Key terms

dollars per million tokens
the price unit that compares an API with your own GPU
break-even volume $N^*$
the monthly tokens where two monthly bills are equal
utilization $u$
the share of hours your GPU is busy
ITL
the wait between two output tokens of one answer

Prerequisites

Take Ranking and re-ranking: multi-objective, diversity, cold start first. It comes right before this lesson in the course.

Where this lesson picks up: You built a ranker that scores and re-ranks every feed. Now the product adds an LLM call to each request, and a new question opens: what does each call cost?

What comes next

Your own GPUs can look cheaper on paper, so the next lesson asks how to test a switch before every user gets it.

Next lesson: A/B design: randomization unit, metrics and guardrails

Take the lesson

Short cards, one idea at a time, with quick checks as you go.