Adam Optimization Algorithm

MediumML Coding

Problem statement

Implement the Adam (Adaptive Moment Estimation) optimizer. Write a function adam_optimizer that starts from x0 and applies Adam updates for a fixed number of iterations, then returns the final parameters.

Parameters:

  • f: the objective function (part of the interface; the updates only need its gradient)
  • grad: a function that returns the gradient of f at a parameter vector
  • x0: initial parameter values (a numpy array)
  • learning_rate: the step size $\alpha$ (default 0.001)
  • beta1: decay rate for the first-moment estimate (default 0.9)
  • beta2: decay rate for the second-moment estimate (default 0.999)
  • epsilon: small constant for numerical stability (default 1e-8)
  • num_iterations: number of update steps (default 10)

Use the standard Adam update with bias correction, and do not modify x0 in place.

Example

Input:

import numpy as np

def objective_function(x):
    return x[0]**2 + x[1]**2

def gradient(x):
    return np.array([2*x[0], 2*x[1]])

x0 = np.array([1.0, 1.0])
print(adam_optimizer(objective_function, gradient, x0))

Output:

[0.99000325 0.99000325]

Reasoning: with bias correction, each early Adam step moves every coordinate by roughly the learning rate in the direction opposite its gradient, so after 10 steps with $\alpha = 0.001$ both coordinates have moved from 1 to about 0.99.

Show hint

Which two exponentially weighted running averages does Adam maintain for every parameter?

Relevant roles: ML Engineer, Deep Learning Engineer, Research Scientist and LLM Engineer.

Related free problems