Debugging a Transformer Text Classifier's Failing Tests

Reported in interviews at OpenAI

Problem statement

How would you debug a text classifier with several failing tests? A PyTorch model fine-tunes a pretrained BERT-style encoder to assign a single class to each customer message. Some bugs are already documented, while others still need diagnosis.

Diagnose the failing tests and show fixes with regression tests.

Show hint

For each failing test, list the causes you can rule out from what the test itself does (mode, padding side, device, precision) before guessing.

Relevant roles: ML Engineer, Deep Learning Engineer and Applied Scientist.

Related free problems