Q/K/V Calculations and FLOPs in Transformer Attention
Problem statement
Consider one self-attention layer of a decoder Transformer with model width $d = 4096$ and $h = 32$ heads of size $d_h = 128$, processing a 2,048-token prompt in one forward pass. The layer has query, key, value and output projection matrices, each $d \times d$. Count a multiply-add as 2 FLOPs, so multiplying an $m \times k$ matrix by a $k \times p$ matrix costs $2mkp$ FLOPs.
Show hint
Write down the shape of every matrix in the computation.
Relevant roles: LLM Engineer, Deep Learning Engineer, ML Engineer and Research Scientist.
Related free problems
- Compute Attention Memory Traffic and FLOPsML CodingMedium