High Training Loss on 20 Examples: Will More Data Fix It?
Problem statement
You are building a neural network for a classification task. As a first check, you train it on just 20 labeled examples. Training converges (the loss stops decreasing), but the final training loss is still very high.
You decide to fix this by training the same network on 10,000 examples instead.
- Is training on more data the right fix for this symptom?
- If you train on the 10,000 examples anyway, what is the most likely result?
- What would you do instead to fix the problem?
Show hint
Which kind of error does the training loss itself measure, and which kind does adding data reduce?
Relevant roles: Deep Learning Engineer, ML Engineer and Applied Scientist.