Estimate KV Cache Size from Model Config

MediumML Coding

Problem statement

Implement a function that estimates the memory footprint of the Key-Value (KV) cache used during autoregressive inference in Transformer-based language models.

During text generation, Transformers cache the Key and Value projections from previous tokens so they don't need to be recomputed at each decoding step. Understanding how large this cache grows is essential for planning GPU memory budgets and choosing sequence length limits.

Function Input:
  • model_config: A dictionary with the following keys:
    • 'num_layers': Number of transformer layers (int)
    • 'num_attention_heads': Number of query attention heads (int)
    • 'num_kv_heads' (optional): Number of key/value heads. If not present, assume it equals num_attention_heads (standard multi-head attention)
    • 'hidden_size': Model hidden dimension (int)
    • 'dtype_bytes': Number of bytes per parameter element (e.g., 2 for FP16, 4 for FP32, 1 for INT8)
  • batch_size: Number of sequences being processed simultaneously (int)
  • seq_len: Number of tokens stored in the cache (int)
Function Output:

Return a dictionary with:

  • 'kv_cache_elements': Total number of scalar elements stored in the KV cache (int)
  • 'kv_cache_size_bytes': Total memory in bytes (int)
  • 'kv_cache_size_mb': Total memory in megabytes (rounded to 4 decimal places), using 1 MB = 1024 * 1024 bytes
  • 'per_layer_size_mb': Memory per layer in megabytes (rounded to 4 decimal places)
  • 'per_token_size_kb': Memory per token (for a single sequence, not multiplied by batch) in kilobytes (rounded to 4 decimal places), using 1 KB = 1024 bytes

Note: The head dimension is derived from the hidden size and the number of query attention heads.

Example

Input:

model_config = {'num_layers': 12, 'num_attention_heads': 12, 'hidden_size': 768, 'dtype_bytes': 2}, batch_size = 1, seq_len = 512

Output:

{'kv_cache_elements': 9437184, 'kv_cache_size_bytes': 18874368, 'kv_cache_size_mb': 18.0, 'per_layer_size_mb': 1.5, 'per_token_size_kb': 36.0}

Reasoning: head_dim = 768 / 12 = 64. Since num_kv_heads is not specified, it defaults to 12 (standard MHA). Total elements = 1 12 2 12 64 512 = 9,437,184. Each element uses 2 bytes (FP16), so total bytes = 18,874,368. Converting: 18,874,368 / (1024^2) = 18.0 MB. Per layer: 18.0 / 12 = 1.5 MB. Per token: 12 layers 2 12 heads 64 dim * 2 bytes = 36,864 bytes = 36.0 KB.

Show hint

What does a transformer need to remember about every previous token in order to attend to it at the next step?

Relevant roles: LLM Engineer, AI Engineer, ML Engineer, Gen AI Engineer and MLOps Engineer.

Related free problems