How Self-Attention Works, From Intuition to Tensor Shapes

MediumLLMs & GenAI

Reported in interviews at Alibaba Capital One Huawei IBM Snap TikTok

Problem statement

How does self-attention work inside a decoder-only language model? Walk through the operations and tensor shapes in a small model trained on batches of sequences. Training uses a causal mask, and the input embedding is tied to the output layer. Explain attention operations, tensor sizes and the resulting weights.

Show hint

Start from the idea of a soft lookup: what is compared with what, and what is averaged?

Relevant roles: LLM Engineer, Deep Learning Engineer, ML Engineer, Research Scientist and Applied Scientist.

Related free problems