Warmup + Cosine Decay Schedule

EasyML Coding

Problem statement

Implement a learning rate schedule that combines linear warmup and cosine decay, and return the learning rate for each of the T training steps (steps are indexed $t = 0, \dots, T-1$).

  • Warmup ($t < W$): the rate rises linearly from 0, $\eta_t = \frac{t}{W}\,\eta_{max}$, so step 0 has rate 0 and step $W$ is the first step at $\eta_{max}$.
  • Cosine decay ($t \ge W$): $\eta_t = \eta_{min} + \frac{1}{2}(\eta_{max} - \eta_{min})\left(1 + \cos\left(\pi \frac{t - W}{T - W}\right)\right)$.

Here $\eta_{max}$ = lr_max and $\eta_{min}$ = lr_min. With W = 0 there is no warmup and step 0 starts at lr_max. Return a list of T floats.

Example

Input:

T=10, W=3, lr_max=1.0, lr_min=0.0

Output:

[0.0, 0.3333, 0.6667, 1.0, 0.9505, 0.8117, 0.6113, 0.3887, 0.1883, 0.0495]

Reasoning: During warmup (steps 0-2), learning rate increases linearly from 0 to 1.0. After warmup, it follows a cosine decay from 1.0 down to 0.0 over the remaining 7 steps.

Show hint

How should the learning rate behave right at the start of training compared with the middle and the end?

Relevant roles: ML Engineer, Deep Learning Engineer, LLM Engineer and Research Scientist.

Related free problems