Detecting Harmful Content in LLM Prompts and Responses

HardLLMs & GenAI

Reported in interviews at Databricks

Problem statement

How would you build a safety layer for a multilingual chat assistant? It must check prompts and streamed responses against written harm categories and severity levels with almost no visible delay. Harmful text must be checked before reaching the screen, while benign questions are already being refused too often.

Choose thresholds, design serving and evaluation, and measure production recall.

Show hint

Before choosing a model, decide what the labels are, what action each label triggers, and at which moments in a chat turn content can be inspected.

Relevant roles: ML Engineer, AI Engineer, LLM Engineer and Gen AI Engineer.

Related free problems