Q/K/V Calculations and FLOPs in Transformer Attention

MediumLLMs & GenAI

Reported in interviews at TikTok

Problem statement

Consider one self-attention layer of a decoder Transformer with model width $d = 4096$ and $h = 32$ heads of size $d_h = 128$, processing a 2,048-token prompt in one forward pass. The layer has query, key, value and output projection matrices, each $d \times d$. Count a multiply-add as 2 FLOPs, so multiplying an $m \times k$ matrix by a $k \times p$ matrix costs $2mkp$ FLOPs.

Show hint

Write down the shape of every matrix in the computation.

Relevant roles: LLM Engineer, Deep Learning Engineer, ML Engineer and Research Scientist.

Related free problems