Glossary: Deep Learning Basics
Vanishing gradients
Also written: vanish
Gradients that shrink toward zero so weights barely change and learning stalls, for example when huge inputs push a softmax to exactly 0 or 1. ReLU's slope of 1 helps avoid it.
Glossary: Deep Learning Basics
Also written: vanish
Gradients that shrink toward zero so weights barely change and learning stalls, for example when huge inputs push a softmax to exactly 0 or 1. ReLU's slope of 1 helps avoid it.