Designing a Multimodal RAG Assistant over Text, PDFs and Screenshots
Problem statement
How would you build a grounded question-answering assistant for hardware engineers? Its changing knowledge base mixes wiki pages, tickets, PDFs, tables, diagrams and screenshots with little image metadata. Answers must arrive quickly and cite the exact passage, page or image used.
Explain image retrieval, revision handling and grounded generation.
Show hint
What does each modality need to become before it can be matched against a natural-language question, and what must you keep so an answer can cite it?
Relevant roles: ML Engineer, AI Engineer, LLM Engineer and Gen AI Engineer.