Language models

Transformer

Picture a group editing a paragraph: everyone reads relevant words, then revises their own notes. A transformer repeats this pattern with attention and a feed-forward network. Position information tells the model how tokens are ordered.

Input3 × 4Attention3 × 4Feed-forward3 × 4Output3 × 4
One block preserves a toy sequence shape of 3 tokens by 4 features; residual paths and norms are omitted.
Input3 × 4Attention3 × 4Feed-forward3 × 4Output3 × 4
One block preserves a toy sequence shape of 3 tokens by 4 features; residual paths and norms are omitted.

How it works

  1. Embed input tokens and supply position information.
  2. In a pre-norm block, normalize the input, apply attention, and add the result to the residual stream.
  3. Normalize again, apply a position-wise feed-forward network, and add another residual connection.
  4. Stack blocks and attach a task head; use a causal mask when predicting the next token.

$$h=x+\operatorname{Attention}(\operatorname{Norm}(x)),\qquad y=h+\operatorname{FFN}(\operatorname{Norm}(h))$$

The knobs that matter

depth
more blocks add capacity and computation
hidden width
sets residual size and much of the parameter budget
feed-forward width
controls expansion inside each position-wise network
position encoding
determines how token order enters the block

What interviewers ask

  • How does a transformer encoder differ from a decoder?
  • Why can transformer training process positions in parallel?
  • Where do residual connections and normalization go in a pre-norm block?
  • How do you estimate parameter count and attention cost?

Common mistakes

  • Forgetting position information when explaining how the model sees order.
  • Saying causal training must run one token at a time because generation does.
  • Assuming attention accounts for most parameters in every transformer configuration.

When to use it

Use transformers for sequences whose positions need to exchange information, including text and image patches. Consider smaller or specialized architectures when data, memory, or latency is tightly limited.

Implement it (12 problems)

Learn it

Interview questions about transformer