RMSProp Optimizer
Problem statement
Implement a single update step of the RMSProp optimizer. Given current parameter values, their gradients, and a cache that stores a moving average of past squared gradients, compute the updated parameters and the updated cache.
For each parameter, first update the cache, $v \leftarrow \beta v + (1 - \beta) g^2$, then update the parameter with the new cache, $\theta \leftarrow \theta - \eta\, g / (\sqrt{v} + \epsilon)$, where $\eta$ = lr, $\beta$ = beta and $\epsilon$ = epsilon. RMSProp adapts the step size per parameter: parameters with consistently large gradients take smaller steps, and parameters with small gradients take larger steps.
Return a tuple (updated_params, updated_cache) of Python lists.
Example
Input:
params=[1.0], grads=[0.1], cache=[0.0], lr=0.1, beta=0.9
Output:
([0.683772], [0.001])
Reasoning: First, update cache: $v_{new} = 0.9 \times 0.0 + 0.1 \times 0.1^2 = 0.001$. Then update parameter: $p_{new} = 1.0 - 0.1 \times \frac{0.1}{\sqrt{0.001} + 10^{-8}} = 1.0 - \frac{0.01}{0.0316228} \approx 0.683772$.
Show hint
What statistic of recent gradients would tell you how large a typical gradient is for each parameter?
Relevant roles: ML Engineer, Deep Learning Engineer, Research Scientist and Applied Scientist.
Related free problems
- Multi-Head AttentionMedium
- Masked Self-AttentionMedium