Skip to content

Glossary: Transformers Basics

Gradient clipping

Also written: clip_grad_norm_

Scaling gradients down whenever their combined size exceeds a set limit, such as 1.0. It guards against occasional huge updates that can destabilize training.

Where the book teaches it

Related terms

AdamW