Estimating KV-Cache Memory and Shrinking It
Problem statement
A team serves a 70B-class decoder LLM with grouped-query attention: 80 layers, 64 query heads that share 8 KV heads, head dimension 128, and keys and values cached in FP16 (2 bytes each). It runs on one node of 8 GPUs with 80 GiB each, and a 128k context means 131,072 tokens. A second team ships a 3B on-device model with 28 layers, 24 query heads sharing 8 KV heads of dimension 128, an FP16 cache, 4-bit weights and a 4,096-token context.
Show hint
Write down what is stored for each token at each layer.
Relevant roles: LLM Engineer, AI Architect, MLOps Engineer and AI Engineer.
Related free problems
- Estimate KV-Cache Memory with GQA and Cross-Layer SharingML CodingMedium
- Estimate KV Cache Size from Model ConfigML CodingMedium