Why Long Context Is Hard for Transformers
Problem statement
How would you extend a decoder transformer to read whole documents? Inputs are much longer than those seen in pretraining, and each attention head stores its own keys and values. Compute, memory and attention quality all become constraints on training and serving.
Explain what makes long context difficult and which techniques address each problem.
Show hint
For each problem, ask whether it grows with the number of tokens, the square of it, or with the positions the model saw during training.
Relevant roles: ML Engineer, AI Engineer, LLM Engineer and Research Scientist.