Causal and Padding Masks in Batched Decoder Attention

MediumLLMs & GenAI

Reported in interviews at Applied Intuition Qualcomm

Problem statement

How would you stop a decoder from attending to future tokens or padding? You are implementing attention directly in PyTorch for batches of sequences with different lengths. Training uses next-token prediction and padding at the end of each sequence. Explain causal and padding masks, their tensor shapes, and how to avoid invalid attention rows.

Show hint

For a query at position $i$, list exactly which key positions it may read once both causality and padding are respected.

Relevant roles: LLM Engineer, Deep Learning Engineer, Research Scientist and ML Engineer.

Related free problems