Why Long Context Is Hard for Transformers

HardLLMs & GenAI

Reported in interviews at Dandy

Problem statement

How would you extend a decoder transformer to read whole documents? Inputs are much longer than those seen in pretraining, and each attention head stores its own keys and values. Compute, memory and attention quality all become constraints on training and serving.

Explain what makes long context difficult and which techniques address each problem.

Show hint

For each problem, ask whether it grows with the number of tokens, the square of it, or with the positions the model saw during training.

Relevant roles: ML Engineer, AI Engineer, LLM Engineer and Research Scientist.

Related free problems