Skip to content

Glossary: Deep Learning Basics

Gradient clipping

Also written: clip_grad_norm_, nn.utils.clip_grad_norm_

Scaling gradients down whenever their combined size (the norm) is too large, to stop exploding gradients in deeper networks. `nn.utils.clip_grad_norm_` does it, is called between `backward` and `step`, and returns the norm before clipping.

Where the book teaches it

Related terms

nanGradient