Estimate KV-Cache Memory with GQA and Cross-Layer Sharing

MediumML Coding

Problem statement

Estimate the total key-value (KV) cache memory footprint for a transformer model under four configurations: standard Multi-Head Attention (MHA), Grouped-Query Attention (GQA), MHA with cross-layer KV sharing, and GQA with cross-layer KV sharing.

Given:

  • batch_size: number of sequences
  • seq_len: context length (tokens cached per sequence)
  • emb_dim: model hidden size (equal to n_heads * head_dim)
  • n_heads: number of attention heads
  • n_kv_groups: number of KV groups used by GQA (1 <= n_kv_groups <= n_heads, and n_heads must be divisible by n_kv_groups)
  • n_layers: total number of transformer layers
  • n_kv_producing_layers: number of layers that actually produce a fresh KV cache; remaining layers reuse cached KV from a producing layer (1 <= n_kv_producing_layers <= n_layers)
  • dtype_bytes: bytes per element (e.g. 2 for bf16/fp16, 4 for fp32)

Return a list of five values: [mha_bytes, gqa_bytes, mha_shared_bytes, gqa_shared_bytes, savings_pct]

where savings_pct is the percentage reduction in bytes of the GQA-with-sharing configuration relative to plain MHA, rounded to 2 decimals. The four byte counts must be integers.

Example

Input:

batch_size=2, seq_len=1024, emb_dim=4096, n_heads=32, n_kv_groups=8, n_layers=32, n_kv_producing_layers=8, dtype_bytes=2

Output:

[1073741824, 268435456, 268435456, 67108864, 93.75]

Reasoning: head_dim = 4096/32 = 128. MHA stores K and V of size emb_dim per layer per token: 2210243240962 = 1,073,741,824 bytes. GQA reduces the per-token dim to n_kv_groupshead_dim = 1024, giving 268,435,456 bytes. Cross-layer sharing reduces the layer factor from 32 to 8, yielding 268,435,456 (MHA+share) and 67,108,864 (GQA+share). Savings of GQA+share vs MHA = 1 - 67108864/1073741824 = 93.75%.

Show hint

Write the KV cache size as a product of factors and ask which factor each optimization changes.

Relevant roles: LLM Engineer, ML Engineer, AI Architect, MLOps Engineer and AI Engineer.

Related free problems