Metrics, L1 vs L2 and an Ablation Study for a Ticket-Urgency NLP Pipeline

MediumClassical ML

Reported in interviews at Microsoft

Problem statement

How would you detect urgent support tickets and decide which pipeline components help? Urgent tickets are rare and the on-call desk has limited review capacity. The pipeline uses text processing, retrieval, reranking and an LLM, while a compact word-feature model serves as a fallback. Choose launch metrics and regularization, then design a controlled ablation study.

Show hint

For each question, ask what the people consuming the output actually see: a queue of top-scored tickets, a memory budget, or a release decision.

Relevant roles: ML Engineer, Applied Scientist and Data Scientist.

Related free problems