Gradient Clipping by Global Norm
Problem statement
Implement gradient clipping by global norm. Given a list of gradient arrays (one flat list per parameter) and a threshold max_norm, compute the global L2 norm over all gradient values combined. If the global norm exceeds the threshold, multiply every gradient value by the same factor so that the global norm becomes max_norm (you may add a tiny epsilon such as 1e-6 to the norm in the denominator); otherwise return the gradients unchanged. Keep the original structure: a list of lists with the same lengths.
Example
Input:
gradients=[[3.0, 4.0], [0.0, 0.0]], max_norm=1.0
Output:
[[0.6, 0.8], [0.0, 0.0]]
Reasoning: The global norm is $\sqrt{3^2 + 4^2 + 0^2 + 0^2} = \sqrt{25} = 5.0$. Since $5.0 > 1.0$, we need to clip. The scaling factor is $\frac{1.0}{5.0} = 0.2$. Each gradient is multiplied by 0.2: $[3.0 \times 0.2, 4.0 \times 0.2] = [0.6, 0.8]$ and $[0.0 \times 0.2, 0.0 \times 0.2] = [0.0, 0.0]$.
Show hint
If all the parameter gradients were concatenated into one long vector, what single length would describe it?
Relevant roles: ML Engineer, Deep Learning Engineer, LLM Engineer and Research Scientist.
Related free problems
- Multi-Head AttentionMedium
- Masked Self-AttentionMedium