Self-Attention and Cross-Entropy From Scratch

MediumLLMs & GenAI

Reported in interviews at TikTok

Problem statement

You are building a small transformer text classifier and the interviewer asks you to write, on a whiteboard, the two pieces you would otherwise import from a library: one head of scaled dot-product self-attention, and the batched multi-class cross-entropy loss. Use these shapes: input activations $X \in \mathbb{R}^{B \times T \times d_{model}}$ (batch $B$, sequence length $T$), projection matrices $W_Q, W_K, W_V \in \mathbb{R}^{d_{model} \times d_k}$, a key mask of shape $[B, T]$ (1 for real tokens, 0 for padding), and classifier logits $z \in \mathbb{R}^{B \times C}$ with labels given either as class indices of shape $[B]$ or as one-hot rows of shape $[B, C]$. The loss is the mean over the batch.

While you write, the interviewer asks follow-up questions about masking, the $1/\sqrt{d_k}$ scaling, numerical stability and the role of the position-wise feed-forward network (FFN) in a transformer block.

  1. Write pseudocode for single-head self-attention with shapes at every step, including the padding mask (and say how a causal mask would change it).
  2. A teammate applies the padding mask by multiplying the attention weights by the 0/1 mask after the softmax. What goes wrong, and what should be done instead?
  3. Why are the logits $QK^\top$ divided by $\sqrt{d_k}$ before the softmax?
  4. Write pseudocode for the batched cross-entropy loss that works for both label formats and does not overflow.
  5. Compute the mean loss for a batch of two: example 1 has logits $[1000, 1002, 999]$ and label index 1; example 2 has logits $[2.0, 0.5, -1.0]$ and one-hot label $[0, 0, 1]$.
  6. What does the FFN contribute to a transformer block that attention does not?
Show hint

For each step of attention, which axis is being reduced, and what shape is left?

Relevant roles: ML Engineer, AI Engineer, Deep Learning Engineer and LLM Engineer.

Related free problems