Detecting and Handling Data Shift in Deployed Models

Reported in interviews at Headspace Nubank

Problem statement

How would you keep a loan fraud model useful after launch? Confirmed fraud can take months to arrive, and you only observe outcomes for approved loans. The team sees input distributions moving and later measures a drop in performance. Explain how to detect shifts, choose a response and account for delayed labels.

Show hint

Separate changes in the inputs, in the base rate and in the input-to-label relationship.

Relevant roles: ML Engineer, MLOps Engineer, Applied Scientist and Data Scientist.

Related free problems