AdamW Optimizer Step
Problem statement
Implement a single update step of the AdamW optimizer for a parameter array $w$ and its gradient $g$. AdamW applies weight decay directly to the parameters, separately from the adaptive gradient update.
Write a function adamw_update(w, g, m, v, t, lr, beta1, beta2, epsilon, weight_decay) that:
- Updates the first moment $m$ and second moment $v$ (exponential moving averages of the gradient and the squared gradient).
- Applies bias correction to $m$ and $v$.
- Applies decoupled weight decay, $w \leftarrow w - \eta \lambda w$, using the learning rate $\eta$ =
lrand $\lambda$ =weight_decay. - Applies the Adam step $w \leftarrow w - \eta\, \hat m_t / (\sqrt{\hat v_t} + \epsilon)$.
- Returns
(w_new, m_new, v_new).
Arguments
w: NumPy array, current parameters.g: NumPy array, gradient (same shape asw).m,v: NumPy arrays, first and second moment estimates (same shape asw).t: integer time step, starting from 1.lr,beta1,beta2,epsilon,weight_decay: floats.
Example
Input:
w = np.array([1.0, 2.0]); g = np.array([0.1, -0.2])
m = np.zeros(2); v = np.zeros(2); t = 1
lr = 0.01; beta1 = 0.9; beta2 = 0.999; epsilon = 1e-8; weight_decay = 0.1
w_new, m_new, v_new = adamw_update(w, g, m, v, t, lr, beta1, beta2, epsilon, weight_decay)
print(np.round(w_new, 4))
Output:
[0.989, 2.008]
Reasoning: weight decay shrinks the parameters by a factor $1 - 0.01 \cdot 0.1 = 0.999$, giving $[0.999, 1.998]$. At $t = 1$ the bias-corrected ratio $\hat m / \sqrt{\hat v}$ equals the sign of the gradient, $[1, -1]$, so the Adam step moves each parameter by $0.01$ against its gradient: $[0.989, 2.008]$.
Show hint
Which two running statistics of the gradient does Adam maintain, and why are they biased early in training?
Relevant roles: ML Engineer, Deep Learning Engineer, LLM Engineer and Research Scientist.
Related free problems
- Multi-Head AttentionMedium
- Masked Self-AttentionMedium