GPU Memory Budget for Serving a 70B Model

MediumLLMs & GenAI

Reported in interviews at NVIDIA

Problem statement

A team must serve a Llama-style 70B decoder (80 layers, grouped-query attention with 8 KV heads of dimension 128) with BF16 weights and a BF16 KV cache on 80 GB GPUs (1 GB = $10^9$ bytes). The target load is 32 concurrent sequences of up to 4,096 tokens each. Assume about 3 GB of transient activation workspace in total, about 3 GB of runtime overhead per GPU, and keep 10% of every GPU free for allocator fragmentation.

Show hint

List the things that must sit in GPU memory while the model serves requests.

Relevant roles: LLM Engineer, MLOps Engineer, AI Architect and ML Engineer.

Related free problems