Language models

GPT

Like finishing a friend’s sentence one piece at a time, a GPT model predicts what token comes next. It uses a stack of causal transformer blocks to turn the prefix into a probability distribution. Repeating this prediction produces a sequence.

Read prefixthe catPredict nextsat: 0.6Choose tokensatExtend prefixthe cat sat
Toy next-token probability: sat has probability 0.6; other tokens share the remaining 0.4.
Read prefixthe catPredict nextsat: 0.6Choose tokensatExtend prefixthe cat sat
Toy next-token probability: sat has probability 0.6; other tokens share the remaining 0.4.

How it works

  1. Tokenize text and form inputs with targets shifted forward by one token.
  2. Run embeddings and position information through causal transformer blocks.
  3. Project hidden states to vocabulary logits and train with next-token cross-entropy.
  4. At generation time, choose a token from the final position’s distribution and append it to the prefix.
  5. Repeat until a stop condition, reusing cached keys and values when available.

$$\mathcal{L}=-\frac{1}{T}\sum_{t=1}^{T}\log p_\theta(x_t\mid x_{<t})$$

The knobs that matter

depth and width
set model capacity and computation
context length
limits how much prefix the model can use
temperature
changes sampling probabilities during generation
weight tying
shares input embedding and output projection weights

What interviewers ask

  • Walk me through training and generating text with a decoder-only model.
  • How would you debug a minimal language model whose training loss looks suspiciously low?
  • Why can training use all positions at once while generation remains sequential?
  • How do weight tying and the vocabulary size affect parameter count?

Common mistakes

  • Using unshifted targets so the model learns to copy the token it already sees.
  • Allowing the causal mask to expose future target tokens.
  • Treating a high next-token probability as evidence that a factual claim is true.

When to use it

Use a decoder-only language model for text generation and tasks expressed as continuation. For a fixed prediction task, compare it with a smaller task-specific model before paying the generation cost.

Implement it (8 problems)

Interview questions about gpt