Detecting Harmful Content in LLM Prompts and Responses
Problem statement
How would you build a safety layer for a multilingual chat assistant? It must check prompts and streamed responses against written harm categories and severity levels with almost no visible delay. Harmful text must be checked before reaching the screen, while benign questions are already being refused too often.
Choose thresholds, design serving and evaluation, and measure production recall.
Show hint
Before choosing a model, decide what the labels are, what action each label triggers, and at which moments in a chat turn content can be inspected.
Relevant roles: ML Engineer, AI Engineer, LLM Engineer and Gen AI Engineer.