Warmup + Cosine Decay Schedule
Problem statement
Implement a learning rate schedule that combines linear warmup and cosine decay, and return the learning rate for each of the T training steps (steps are indexed $t = 0, \dots, T-1$).
- Warmup ($t < W$): the rate rises linearly from 0, $\eta_t = \frac{t}{W}\,\eta_{max}$, so step 0 has rate 0 and step $W$ is the first step at $\eta_{max}$.
- Cosine decay ($t \ge W$): $\eta_t = \eta_{min} + \frac{1}{2}(\eta_{max} - \eta_{min})\left(1 + \cos\left(\pi \frac{t - W}{T - W}\right)\right)$.
Here $\eta_{max}$ = lr_max and $\eta_{min}$ = lr_min. With W = 0 there is no warmup and step 0 starts at lr_max. Return a list of T floats.
Example
Input:
T=10, W=3, lr_max=1.0, lr_min=0.0
Output:
[0.0, 0.3333, 0.6667, 1.0, 0.9505, 0.8117, 0.6113, 0.3887, 0.1883, 0.0495]
Reasoning: During warmup (steps 0-2), learning rate increases linearly from 0 to 1.0. After warmup, it follows a cosine decay from 1.0 down to 0.0 over the remaining 7 steps.
Show hint
How should the learning rate behave right at the start of training compared with the middle and the end?
Relevant roles: ML Engineer, Deep Learning Engineer, LLM Engineer and Research Scientist.
Related free problems
- Multi-Head AttentionMedium
- Masked Self-AttentionMedium