High Offline Accuracy but Strange Production Behavior

Reported in interviews at Bridgewater

Problem statement

A risk classifier validates well but disappoints in live review. Reliable labels arrive late, and reviewers see only the highest scores. You have training and inference code, model artifacts and request logs. What would you check first, how would you measure live quality, and how would you roll out a fix?

Show hint

What evidence would distinguish a transformation error from a change in the underlying problem?

Relevant roles: ML Engineer, MLOps Engineer, Data Scientist and AI Architect.

Related free problems