Evaluating and Debugging a Production RAG Search System

HardLLMs & GenAI

Reported in interviews at Plaid

Problem statement

How would you evaluate and debug a question-answering assistant in a financial services app? It retrieves help articles, policies and account records, then writes answers with citations. Documents change often, unsupported answers carry serious risk, and access controls must keep each user’s data private.

Explain retrieval and answer metrics, debugging and evaluation datasets.

Show hint

Ask what the generator actually sees, and which metric measures exactly that rather than a stage upstream of it.

Relevant roles: ML Engineer, AI Engineer, LLM Engineer and Gen AI Engineer.

Related free problems