GPU Memory Budget for Serving a 70B Model
Problem statement
A team must serve a Llama-style 70B decoder (80 layers, grouped-query attention with 8 KV heads of dimension 128) with BF16 weights and a BF16 KV cache on 80 GB GPUs (1 GB = $10^9$ bytes). The target load is 32 concurrent sequences of up to 4,096 tokens each. Assume about 3 GB of transient activation workspace in total, about 3 GB of runtime overhead per GPU, and keep 10% of every GPU free for allocator fragmentation.
Show hint
List the things that must sit in GPU memory while the model serves requests.
Relevant roles: LLM Engineer, MLOps Engineer, AI Architect and ML Engineer.