GELU
Also written: nn.GELU
A smooth activation function used in GPT models. Like ReLU it passes large positive values and blocks large negative ones, but it bends smoothly around zero.
Also written: nn.GELU
A smooth activation function used in GPT models. Like ReLU it passes large positive values and blocks large negative ones, but it bends smoothly around zero.