RAG vs Fine-Tuning for Knowledge Assistants

MediumLLMs & GenAI

Reported in interviews at Glean

Problem statement

A team runs a chatbot that answers questions across a broad product and policy domain using retrieval-augmented generation (RAG) with prompting. A second team is building an internal assistant for an enterprise whose documents live in many tools, change daily and are visible only to some employees, and whose deal records sit in a structured database. Both teams must decide what to solve with retrieval and what with fine-tuning, and know where RAG breaks.

Show hint

Separate what the model must know (facts) from how it must behave (format, style, skills).

Relevant roles: AI Engineer, Gen AI Engineer, AI Architect and LLM Engineer.

Related free problems