Designing a Multimodal RAG Assistant over Text, PDFs and Screenshots

HardLLMs & GenAI

Reported in interviews at Apple

Problem statement

How would you build a grounded question-answering assistant for hardware engineers? Its changing knowledge base mixes wiki pages, tickets, PDFs, tables, diagrams and screenshots with little image metadata. Answers must arrive quickly and cite the exact passage, page or image used.

Explain image retrieval, revision handling and grounded generation.

Show hint

What does each modality need to become before it can be matched against a natural-language question, and what must you keep so an answer can cite it?

Relevant roles: ML Engineer, AI Engineer, LLM Engineer and Gen AI Engineer.

Related free problems