Pre-norm
Placing layer norm at the input of each part, x + f(LayerNorm(x)), instead of after the addition as in the 2017 paper. It trains more stably in deep stacks and is now standard.
Placing layer norm at the input of each part, x + f(LayerNorm(x)), instead of after the addition as in the 2017 paper. It trains more stably in deep stacks and is now standard.