Glossary: Deep Learning Basics
ReLU
Also written: rectified linear unit, nn.ReLU
The activation max(0, z), the default between layers. Its slope is 1 for positive inputs, so gradients pass through many layers without shrinking.
Glossary: Deep Learning Basics
Also written: rectified linear unit, nn.ReLU
The activation max(0, z), the default between layers. Its slope is 1 for positive inputs, so gradients pass through many layers without shrinking.