Language models
GPT
Like finishing a friend’s sentence one piece at a time, a GPT model predicts what token comes next. It uses a stack of causal transformer blocks to turn the prefix into a probability distribution. Repeating this prediction produces a sequence.
How it works
- Tokenize text and form inputs with targets shifted forward by one token.
- Run embeddings and position information through causal transformer blocks.
- Project hidden states to vocabulary logits and train with next-token cross-entropy.
- At generation time, choose a token from the final position’s distribution and append it to the prefix.
- Repeat until a stop condition, reusing cached keys and values when available.
$$\mathcal{L}=-\frac{1}{T}\sum_{t=1}^{T}\log p_\theta(x_t\mid x_{<t})$$
The knobs that matter
- depth and width
- set model capacity and computation
- context length
- limits how much prefix the model can use
- temperature
- changes sampling probabilities during generation
- weight tying
- shares input embedding and output projection weights
What interviewers ask
- Walk me through training and generating text with a decoder-only model.
- How would you debug a minimal language model whose training loss looks suspiciously low?
- Why can training use all positions at once while generation remains sequential?
- How do weight tying and the vocabulary size affect parameter count?
Common mistakes
- Using unshifted targets so the model learns to copy the token it already sees.
- Allowing the causal mask to expose future target tokens.
- Treating a high next-token probability as evidence that a factual claim is true.
When to use it
Use a decoder-only language model for text generation and tasks expressed as continuation. For a fixed prediction task, compare it with a smaller task-specific model before paying the generation cost.