LLM Serving: Latency vs Throughput and Batched Inference
Problem statement
How would you balance response speed and throughput when serving a language model? Batching shares the cost of reading model weights, but requests have different prompt and output lengths. Both GPU compute and memory can limit serving. Estimate the batching limit, explain the serving trade-offs, and choose a scheduler.
Show hint
Compare the bytes a decode step must read with the FLOPs it performs, and see how each scales with batch size.
Relevant roles: LLM Engineer, MLOps Engineer, AI Architect and AI Engineer.