Momentum Optimizer
Problem statement
Implement one update step of gradient descent with momentum. Given the current parameter, its gradient grad, and the current velocity, compute
$$v_t = \gamma\, v_{t-1} + \eta\, \nabla_\theta J(\theta), \qquad \theta_t = \theta_{t-1} - v_t$$
where $\gamma$ is momentum (default 0.9) and $\eta$ is learning_rate (default 0.01). The function must work for both scalars and NumPy arrays. Return (updated_parameter, updated_velocity), each rounded to 5 decimal places.
Example
Input:
momentum_optimizer(1.0, 0.1, 0.1)
Output:
(0.909, 0.091)
Reasoning: $v = 0.9 \times 0.1 + 0.01 \times 0.1 = 0.091$ and $\theta = 1.0 - 0.091 = 0.909$.
Show hint
What should the new velocity remember from the previous one, and what should it add?
Relevant roles: ML Engineer, Deep Learning Engineer, Research Scientist and Applied Scientist.
Related free problems
- Multi-Head AttentionMedium
- Masked Self-AttentionMedium