Glossary: Deep Learning Basics

zero_grad

Also written: opt.zero_grad()

Clears the old gradients before each backward pass. Forgetting it makes gradients pile up and the loss rise.