Momentum Optimizer

EasyML Coding

Problem statement

Implement one update step of gradient descent with momentum. Given the current parameter, its gradient grad, and the current velocity, compute

$$v_t = \gamma\, v_{t-1} + \eta\, \nabla_\theta J(\theta), \qquad \theta_t = \theta_{t-1} - v_t$$

where $\gamma$ is momentum (default 0.9) and $\eta$ is learning_rate (default 0.01). The function must work for both scalars and NumPy arrays. Return (updated_parameter, updated_velocity), each rounded to 5 decimal places.

Example

Input:

momentum_optimizer(1.0, 0.1, 0.1)

Output:

(0.909, 0.091)

Reasoning: $v = 0.9 \times 0.1 + 0.01 \times 0.1 = 0.091$ and $\theta = 1.0 - 0.091 = 0.909$.

Show hint

What should the new velocity remember from the previous one, and what should it add?

Relevant roles: ML Engineer, Deep Learning Engineer, Research Scientist and Applied Scientist.

Related free problems