Parameter Count and Attention Share of a 64-Layer Transformer
Problem statement
A decoder-only transformer has hidden size $D = 4096$, feed-forward width $F = 4D$, vocabulary size $V = 32{,}000$ and $L = 64$ layers. Each layer uses multi-head attention in which $D$ is split across $N$ heads of size $H$ ($NH = D$, no grouped or shared key-value heads), and keys and values are cached in int8 at inference time. The feed-forward block is gated (SwiGLU-style): two input projections from $D$ to $F$ and one output projection from $F$ back to $D$. The input embedding and the output unembedding are separate $V \times D$ matrices. Ignore biases and normalization parameters.
- How many parameters does the model have?
- What fraction of them are attention parameters?
- Does caching keys and values in int8 change your count?
Show hint
Which weight matrices exist in one layer, and what are their shapes in terms of $D$ and $F$?
Relevant roles: LLM Engineer, Deep Learning Engineer and ML Engineer.