Evaluating RAG Systems, RAG Components and the Transformer

MediumLLMs & GenAI

Reported in interviews at Cohere

Problem statement

How would you evaluate an assistant that answers employee questions from internal documents? It retrieves passages, gives them to a model and cites sources in its answers. You need to tell retrieval failures from answer-writing failures and evaluate a new customer deployment with no labeled data.

Explain stage-level evaluation and how to test quality when labels are missing.

Show hint

Picture the pipeline as stages and ask where each kind of failure would originate.

Relevant roles: AI Engineer, Gen AI Engineer, LLM Engineer, AI Architect and ML Engineer.

Related free problems