Self-Attention, LoRA, Optimizers and ViT Patch Size
Problem statement
How would you explain attention and the memory cost of fine-tuning a language model? The team can update all model weights or train small adapters. GPU memory is limited, so the choice affects whether training fits. Explain self-attention, count adapter parameters and compare training memory.
Show hint
For each mechanism, ask what would go wrong without it: unscaled logits, a full-rank update.
Relevant roles: ML Engineer, Deep Learning Engineer, Applied Scientist, Research Scientist and LLM Engineer.
Related free problems
- Memory Savings of LoRA vs Full Fine-Tuning with AdamML CodingEasy