Training Multimodal LLMs and Reasoning Models Versus Chat Models
Problem statement
How would you train an assistant to answer questions about photos, screenshots and scans? A pretrained vision encoder feeds a decoder language model through a small projector, first learning from image captions and then visual conversations. The team must also choose how to handle requests that need several reasoning steps.
Diagnose weak image use, examine multimodal training challenges, and compare reasoning training.
Show hint
For the multimodal part, compare how many tokens an image contributes with how many tokens of supervision a caption provides, and ask what a strong text-only model can get away with.
Relevant roles: ML Engineer, LLM Engineer, Gen AI Engineer, Research Scientist and Applied Scientist.