KV Cache Size Estimator for MLA vs MHA vs GQA
Problem statement
Estimate the KV cache memory footprint (in bytes) for three attention variants used in transformer inference: standard Multi-Head Attention (MHA), Grouped-Query Attention (GQA), and Multi-head Latent Attention (MLA).
Given the batch size, sequence length, number of layers, number of heads, head dimension, number of KV groups (for GQA), MLA latent dimension, attention type, and bytes per element, return the total KV cache size in bytes as an integer.
Formulas:
- MHA: bytes = 2 batch seqlen n_layers n_heads head_dim bytes_per_elem
- GQA: bytes = 2 batch seqlen n_layers n_kv_groups head_dim bytes_per_elem
- MLA: bytes = batch seqlen n_layers latent_dim bytes_per_elem
The factor of 2 in MHA/GQA accounts for storing both K and V; MLA stores a single compressed latent vector per token per layer.
The attn_type argument is one of 'mha', 'gqa', or 'mla'. Return an int.
Example
Input:
batch = 1, seqlen = 1024, n_layers = 32, n_heads = 32, head_dim = 128,
n_kv_groups = 8, latent_dim = 512, attn_type = 'mla', bytes_per_elem = 2
Output:
33554432
Reasoning: For MLA: 1 1024 32 512 2 = 33,554,432 bytes (32 MiB). The same model with MHA would need 2 1 1024 32 32 128 2 = 536,870,912 bytes.
Show hint
For each attention variant, what exactly is stored for one token in one layer?
Relevant roles: LLM Engineer, MLOps Engineer, AI Architect, AI Engineer and Gen AI Engineer.
Related free problems
- Estimate KV-Cache Memory with GQA and Cross-Layer SharingML CodingMedium
- Estimate KV Cache Size from Model ConfigML CodingMedium