Neural networks

Optimizers

Imagine walking downhill in fog, using the slope under your feet to choose each step. An optimizer uses loss gradients to move model weights. Momentum remembers earlier directions, while adaptive methods also track gradient size to scale updates.

Current weightw = 2Gradientg = 3SGD update2 − 0.1 × 3 = 1.7
One SGD step with learning rate 0.1 moves the weight against the gradient.
Current weightw = 2Gradientg = 3SGD update2 − 0.1 × 3 = 1.7
One SGD step with learning rate 0.1 moves the weight against the gradient.

How it works

  1. Run the model on a batch and compute the training loss.
  2. Backpropagate to obtain the gradient of the loss with respect to each parameter.
  3. Use the current gradient and any stored optimizer state to calculate an update.
  4. Apply the update, clear accumulated gradients when appropriate and advance the learning-rate schedule.

$$\theta_{t+1}=\theta_t-\eta_t\nabla_\theta L_t(\theta_t)$$

The knobs that matter

learning rate
controls the overall size of each update
momentum or betas
control how long past gradients and squared gradients are remembered
weight decay
shrinks parameters; AdamW applies this separately from gradient adaptation
schedule
changes the learning rate over training

What interviewers ask

  • How do SGD with momentum and Adam differ?
  • What is the difference between Adam and AdamW?
  • How would you diagnose a learning rate that is too high?
  • Why might you use warmup followed by a decaying learning rate?

Common mistakes

  • Forgetting to clear gradients can unintentionally sum gradients across batches.
  • L2 penalties and decoupled weight decay are not generally equivalent under adaptive updates.
  • A faster drop in training loss does not guarantee better validation performance.

When to use it

Use optimizers whenever you train differentiable models, choosing the update rule and learning rate together. Compare validation behavior and compute cost rather than assuming one optimizer wins on every task.

Implement it (5 problems)

Learn it

Interview questions about optimizers