Adam Optimization Algorithm
Problem statement
Implement the Adam (Adaptive Moment Estimation) optimizer. Write a function adam_optimizer that starts from x0 and applies Adam updates for a fixed number of iterations, then returns the final parameters.
Parameters:
f: the objective function (part of the interface; the updates only need its gradient)grad: a function that returns the gradient offat a parameter vectorx0: initial parameter values (a numpy array)learning_rate: the step size $\alpha$ (default 0.001)beta1: decay rate for the first-moment estimate (default 0.9)beta2: decay rate for the second-moment estimate (default 0.999)epsilon: small constant for numerical stability (default 1e-8)num_iterations: number of update steps (default 10)
Use the standard Adam update with bias correction, and do not modify x0 in place.
Example
Input:
import numpy as np
def objective_function(x):
return x[0]**2 + x[1]**2
def gradient(x):
return np.array([2*x[0], 2*x[1]])
x0 = np.array([1.0, 1.0])
print(adam_optimizer(objective_function, gradient, x0))
Output:
[0.99000325 0.99000325]
Reasoning: with bias correction, each early Adam step moves every coordinate by roughly the learning rate in the direction opposite its gradient, so after 10 steps with $\alpha = 0.001$ both coordinates have moved from 1 to about 0.99.
Show hint
Which two exponentially weighted running averages does Adam maintain for every parameter?
Relevant roles: ML Engineer, Deep Learning Engineer, Research Scientist and LLM Engineer.
Related free problems
- Multi-Head AttentionMedium
- Masked Self-AttentionMedium