Glossary: Deep Learning Basics

Momentum

Also written: SGD with momentum

Each SGD step also carries part (e.g. 90%) of the previous step, like a ball rolling downhill, speeding through long gentle slopes.