AdamW Optimizer Step

EasyML Coding

Problem statement

Implement a single update step of the AdamW optimizer for a parameter array $w$ and its gradient $g$. AdamW applies weight decay directly to the parameters, separately from the adaptive gradient update.

Write a function adamw_update(w, g, m, v, t, lr, beta1, beta2, epsilon, weight_decay) that:

  • Updates the first moment $m$ and second moment $v$ (exponential moving averages of the gradient and the squared gradient).
  • Applies bias correction to $m$ and $v$.
  • Applies decoupled weight decay, $w \leftarrow w - \eta \lambda w$, using the learning rate $\eta$ = lr and $\lambda$ = weight_decay.
  • Applies the Adam step $w \leftarrow w - \eta\, \hat m_t / (\sqrt{\hat v_t} + \epsilon)$.
  • Returns (w_new, m_new, v_new).

Arguments

  • w: NumPy array, current parameters.
  • g: NumPy array, gradient (same shape as w).
  • m, v: NumPy arrays, first and second moment estimates (same shape as w).
  • t: integer time step, starting from 1.
  • lr, beta1, beta2, epsilon, weight_decay: floats.

Example

Input:

w = np.array([1.0, 2.0]); g = np.array([0.1, -0.2])
m = np.zeros(2); v = np.zeros(2); t = 1
lr = 0.01; beta1 = 0.9; beta2 = 0.999; epsilon = 1e-8; weight_decay = 0.1
w_new, m_new, v_new = adamw_update(w, g, m, v, t, lr, beta1, beta2, epsilon, weight_decay)
print(np.round(w_new, 4))

Output:

[0.989, 2.008]

Reasoning: weight decay shrinks the parameters by a factor $1 - 0.01 \cdot 0.1 = 0.999$, giving $[0.999, 1.998]$. At $t = 1$ the bias-corrected ratio $\hat m / \sqrt{\hat v}$ equals the sign of the gradient, $[1, -1]$, so the Adam step moves each parameter by $0.01$ against its gradient: $[0.989, 2.008]$.

Show hint

Which two running statistics of the gradient does Adam maintain, and why are they biased early in training?

Relevant roles: ML Engineer, Deep Learning Engineer, LLM Engineer and Research Scientist.

Related free problems