Estimating KV-Cache Memory and Shrinking It

MediumLLMs & GenAI

Reported in interviews at xAI Apple

Problem statement

A team serves a 70B-class decoder LLM with grouped-query attention: 80 layers, 64 query heads that share 8 KV heads, head dimension 128, and keys and values cached in FP16 (2 bytes each). It runs on one node of 8 GPUs with 80 GiB each, and a 128k context means 131,072 tokens. A second team ships a 3B on-device model with 28 layers, 24 query heads sharing 8 KV heads of dimension 128, an FP16 cache, 4-bit weights and a 4,096-token context.

Show hint

Write down what is stored for each token at each layer.

Relevant roles: LLM Engineer, AI Architect, MLOps Engineer and AI Engineer.

Related free problems