KV Cache Estimator with Sliding Window Attention
Problem statement
Estimate the total key-value (KV) cache memory in bytes for a transformer that mixes full attention layers and sliding-window attention (SWA) layers. With grouped-query attention (GQA) the cache stores num_kv_heads per layer (not num_heads). For full-attention layers, every token's K and V are kept (effective sequence length = seqlen). For SWA layers, only the most recent window tokens are retained (effective sequence length = min(seqlen, window)).
Implement kv_cache_bytes(num_layers_full, num_layers_swa, batch, seqlen, window, num_kv_heads, head_dim, dtype_bytes) that returns the total KV cache size in bytes as an integer.
For a single layer, KV cache bytes = 2 * batch * effective_seqlen * num_kv_heads * head_dim * dtype_bytes (the factor of 2 accounts for storing both K and V). Sum across all layers.
Example
Input:
num_layers_full=2, num_layers_swa=2, batch=1, seqlen=1024, window=128, num_kv_heads=4, head_dim=64, dtype_bytes=2
Output:
2359296
Reasoning: Per-token-per-layer cost = 2 1 4 64 2 = 1024 bytes. Full layers store 1024 tokens: 2 1024 1024 = 2097152 bytes. SWA layers store min(1024,128)=128 tokens: 2 128 1024 = 262144 bytes. Total = 2097152 + 262144 = 2359296 bytes.
Show hint
What is the memory cost of caching a single token in a single layer, and which head count does grouped-query attention actually store?
Relevant roles: LLM Engineer, ML Engineer, AI Engineer, MLOps Engineer and AI Architect.
Related free problems
- Multi-Head AttentionMedium
- Masked Self-AttentionMedium