Compute Attention Memory Traffic and FLOPs
Problem statement
Implement a function that computes the memory traffic (in bytes) and floating-point operations (FLOPs) for a standard (unfused) self-attention forward pass.
The standard self-attention mechanism consists of three sequential steps:
- Score computation: S = Q @ K^T
- Normalization: P = softmax(S)
- Output: O = P @ V
where Q, K, V each have shape (B, h, N, d) for batch size B, number of heads h, sequence length N, and head dimension d.
Your function should compute:
- The FLOPs for each of the three steps separately
- The total FLOPs
- The total memory traffic in bytes, assuming each intermediate result (S, P) is fully materialized (written to and read from memory) between steps
- The arithmetic intensity (total FLOPs divided by total memory bytes)
For FLOPs counting:
- A matrix multiplication of (M x K) @ (K x P) costs 2MK*P FLOPs (one multiply and one add per output element per inner dimension step)
- Softmax over each row of length N costs 5*N operations (find max, subtract max, exponentiate, sum, divide)
For memory traffic, count all reads and writes for each step assuming no kernel fusion (each step reads its inputs from memory and writes its output to memory).
Return a dictionary with keys: 'qk_flops', 'softmax_flops', 'pv_flops', 'total_flops', 'memory_bytes', and 'arithmetic_intensity' (rounded to 2 decimal places).
Example
Input:
attention_memory_flops(B=1, h=1, N=4, d=2, bytes_per_element=2)
Output:
{'qk_flops': 64, 'softmax_flops': 80, 'pv_flops': 64, 'total_flops': 208, 'memory_bytes': 192, 'arithmetic_intensity': 1.08}
Reasoning: With B=1, h=1, N=4, d=2:
- Q@K^T FLOPs: 211162 = 64
- Softmax FLOPs: 511*16 = 80
- P@V FLOPs: 211162 = 64
- Total FLOPs: 208
- Memory elements: Step 1 reads Q(42=8) + K(8) and writes S(16) = 32; Step 2 reads S(16) and writes P(16) = 32; Step 3 reads P(16) + V(8) and writes O(8) = 32. Total = 96 elements 2 bytes = 192 bytes
- Arithmetic intensity: 208/192 = 1.08 FLOPs/byte
Show hint
Break the forward pass into its three stages and ask which tensors each stage reads and writes.
Relevant roles: LLM Engineer, ML Engineer, AI Engineer, Deep Learning Engineer and Gen AI Engineer.
Related free problems
- Q/K/V Calculations and FLOPs in Transformer AttentionLLMs & GenAIMedium