Neural networks
Optimizers
Imagine walking downhill in fog, using the slope under your feet to choose each step. An optimizer uses loss gradients to move model weights. Momentum remembers earlier directions, while adaptive methods also track gradient size to scale updates.
How it works
- Run the model on a batch and compute the training loss.
- Backpropagate to obtain the gradient of the loss with respect to each parameter.
- Use the current gradient and any stored optimizer state to calculate an update.
- Apply the update, clear accumulated gradients when appropriate and advance the learning-rate schedule.
$$\theta_{t+1}=\theta_t-\eta_t\nabla_\theta L_t(\theta_t)$$
The knobs that matter
- learning rate
- controls the overall size of each update
- momentum or betas
- control how long past gradients and squared gradients are remembered
- weight decay
- shrinks parameters; AdamW applies this separately from gradient adaptation
- schedule
- changes the learning rate over training
What interviewers ask
- How do SGD with momentum and Adam differ?
- What is the difference between Adam and AdamW?
- How would you diagnose a learning rate that is too high?
- Why might you use warmup followed by a decaying learning rate?
Common mistakes
- Forgetting to clear gradients can unintentionally sum gradients across batches.
- L2 penalties and decoupled weight decay are not generally equivalent under adaptive updates.
- A faster drop in training loss does not guarantee better validation performance.
When to use it
Use optimizers whenever you train differentiable models, choosing the update rule and learning rate together. Compare validation behavior and compute cost rather than assuming one optimizer wins on every task.
Implement it (5 problems)
- AdamW Optimizer StepFreeEasySolve
- Momentum OptimizerFreeEasySolve
- RMSProp OptimizerFreeEasySolve
- Adam Optimization AlgorithmFreeMediumSolve