Language models
Transformer
Picture a group editing a paragraph: everyone reads relevant words, then revises their own notes. A transformer repeats this pattern with attention and a feed-forward network. Position information tells the model how tokens are ordered.
How it works
- Embed input tokens and supply position information.
- In a pre-norm block, normalize the input, apply attention, and add the result to the residual stream.
- Normalize again, apply a position-wise feed-forward network, and add another residual connection.
- Stack blocks and attach a task head; use a causal mask when predicting the next token.
$$h=x+\operatorname{Attention}(\operatorname{Norm}(x)),\qquad y=h+\operatorname{FFN}(\operatorname{Norm}(h))$$
The knobs that matter
- depth
- more blocks add capacity and computation
- hidden width
- sets residual size and much of the parameter budget
- feed-forward width
- controls expansion inside each position-wise network
- position encoding
- determines how token order enters the block
What interviewers ask
- How does a transformer encoder differ from a decoder?
- Why can transformer training process positions in parallel?
- Where do residual connections and normalization go in a pre-norm block?
- How do you estimate parameter count and attention cost?
Common mistakes
- Forgetting position information when explaining how the model sees order.
- Saying causal training must run one token at a time because generation does.
- Assuming attention accounts for most parameters in every transformer configuration.
When to use it
Use transformers for sequences whose positions need to exchange information, including text and image patches. Consider smaller or specialized architectures when data, memory, or latency is tightly limited.
Implement it (12 problems)
- Positional Encoding CalculatorFreeMediumSolve