Glossary: Deep Learning Basics
Layer normalization
Also written: LayerNorm, nn.LayerNorm
Standardizes each example across its own features. It doesn't depend on the batch, so it behaves the same in training and evaluation, and transformers use it.
Glossary: Deep Learning Basics
Also written: LayerNorm, nn.LayerNorm
Standardizes each example across its own features. It doesn't depend on the batch, so it behaves the same in training and evaluation, and transformers use it.