Parameter Count and Attention Share of a 64-Layer Transformer

MediumLLMs & GenAI

Reported in interviews at OpenAI

Problem statement

A decoder-only transformer has hidden size $D = 4096$, feed-forward width $F = 4D$, vocabulary size $V = 32{,}000$ and $L = 64$ layers. Each layer uses multi-head attention in which $D$ is split across $N$ heads of size $H$ ($NH = D$, no grouped or shared key-value heads), and keys and values are cached in int8 at inference time. The feed-forward block is gated (SwiGLU-style): two input projections from $D$ to $F$ and one output projection from $F$ back to $D$. The input embedding and the output unembedding are separate $V \times D$ matrices. Ignore biases and normalization parameters.

  1. How many parameters does the model have?
  2. What fraction of them are attention parameters?
  3. Does caching keys and values in int8 change your count?
Show hint

Which weight matrices exist in one layer, and what are their shapes in terms of $D$ and $F$?

Relevant roles: LLM Engineer, Deep Learning Engineer and ML Engineer.

Related free problems