ML Fundamentals Through a Watch-Prediction Model
Problem statement
You join the team that predicts whether a user will start watching a title shown on a streaming app's home page. Each training row is one (user, title) impression with tabular features (user tenure, recent watch counts by genre, title age, popularity, device, hour) and a label: 1 if the user started the title within the session. About 2% of impressions are positives. The interviewer walks through the fundamentals using this project: tree ensembles, how the model is trained and validated, how it is evaluated, title embeddings, and transformer basics.
- Tree ensembles. A random forest of 500 fully grown trees reaches 0.80 validation AUC (training AUC 0.97). Growing it to 2,000 trees leaves validation AUC at 0.80, while a gradient-boosted model with 600 depth-4 trees and a learning rate of 0.05 reaches 0.84. Why does adding trees not help the forest, and what does boosting do differently?
- Training and validation. A teammate split the impression rows randomly 80/10/10 into train, validation and test sets drawn from the same 8 weeks. The validation AUC is 0.86, but on the week after the data ends the model scores 0.74. What is the most likely cause?
- Evaluation. On 100,000 validation impressions (2,000 positives), a threshold of 0.5 flags 1,500 impressions, of which 900 are positives. What is the F1 score? Why is accuracy a poor summary here?
- Embeddings. The model currently encodes the 50,000 title IDs as one-hot features. A teammate proposes learned title embeddings of dimension 64. What is the main benefit?
- Transformers. Explain what self-attention computes, why transformers replaced RNNs for sequence modeling, why they need positional information, and how the cost grows with sequence length.
Show hint
For the ensembles, ask which component of the error averaging can remove and which it cannot.
Relevant roles: ML Engineer, Data Scientist and Applied Scientist.