Self-Attention, LoRA, Optimizers and ViT Patch Size

MediumDeep Learning

Reported in interviews at Netflix

Problem statement

How would you explain attention and the memory cost of fine-tuning a language model? The team can update all model weights or train small adapters. GPU memory is limited, so the choice affects whether training fits. Explain self-attention, count adapter parameters and compare training memory.

Show hint

For each mechanism, ask what would go wrong without it: unscaled logits, a full-rank update.

Relevant roles: ML Engineer, Deep Learning Engineer, Applied Scientist, Research Scientist and LLM Engineer.

Related free problems