Evaluating and Debugging a Production RAG Search System
Problem statement
How would you evaluate and debug a question-answering assistant in a financial services app? It retrieves help articles, policies and account records, then writes answers with citations. Documents change often, unsupported answers carry serious risk, and access controls must keep each user’s data private.
Explain retrieval and answer metrics, debugging and evaluation datasets.
Show hint
Ask what the generator actually sees, and which metric measures exactly that rather than a stage upstream of it.
Relevant roles: ML Engineer, AI Engineer, LLM Engineer and Gen AI Engineer.