Vision Encoders and LLM Inference Bottlenecks in a Photo Assistant

HardLLMs & GenAI

Reported in interviews at Apple

Problem statement

How would you build an assistant that answers questions about photos? A vision encoder feeds image tokens through a small projector into a decoder-only language model. GPU memory and bandwidth constrain how many conversations you can serve and how quickly they respond.

Choose the encoder and explain inference memory, concurrency and speed.

Show hint

What does the language model need from an image in order to attend to it the way it attends to words?

Relevant roles: ML Engineer, AI Engineer and LLM Engineer.

Related free problems