Estimate KV-Cache Memory with GQA and Cross-Layer Sharing
Problem statement
Estimate the total key-value (KV) cache memory footprint for a transformer model under four configurations: standard Multi-Head Attention (MHA), Grouped-Query Attention (GQA), MHA with cross-layer KV sharing, and GQA with cross-layer KV sharing.
Given:
batch_size: number of sequencesseq_len: context length (tokens cached per sequence)emb_dim: model hidden size (equal ton_heads * head_dim)n_heads: number of attention headsn_kv_groups: number of KV groups used by GQA (1 <= n_kv_groups <= n_heads, and n_heads must be divisible by n_kv_groups)n_layers: total number of transformer layersn_kv_producing_layers: number of layers that actually produce a fresh KV cache; remaining layers reuse cached KV from a producing layer (1 <= n_kv_producing_layers <= n_layers)dtype_bytes: bytes per element (e.g. 2 for bf16/fp16, 4 for fp32)
Return a list of five values:
[mha_bytes, gqa_bytes, mha_shared_bytes, gqa_shared_bytes, savings_pct]
where savings_pct is the percentage reduction in bytes of the GQA-with-sharing configuration relative to plain MHA, rounded to 2 decimals. The four byte counts must be integers.
Example
Input:
batch_size=2, seq_len=1024, emb_dim=4096, n_heads=32, n_kv_groups=8, n_layers=32, n_kv_producing_layers=8, dtype_bytes=2
Output:
[1073741824, 268435456, 268435456, 67108864, 93.75]
Reasoning: head_dim = 4096/32 = 128. MHA stores K and V of size emb_dim per layer per token: 2210243240962 = 1,073,741,824 bytes. GQA reduces the per-token dim to n_kv_groupshead_dim = 1024, giving 268,435,456 bytes. Cross-layer sharing reduces the layer factor from 32 to 8, yielding 268,435,456 (MHA+share) and 67,108,864 (GQA+share). Savings of GQA+share vs MHA = 1 - 67108864/1073741824 = 93.75%.
Show hint
Write the KV cache size as a product of factors and ask which factor each optimization changes.
Relevant roles: LLM Engineer, ML Engineer, AI Architect, MLOps Engineer and AI Engineer.
Related free problems
- Estimating KV-Cache Memory and Shrinking ItLLMs & GenAIMedium
- Multi-Head AttentionMedium