SwiGLU
A gated MLP used in Llama-style models, where a learned gate multiplies two branches. It gives better quality for the same compute than a plain GELU MLP.
A gated MLP used in Llama-style models, where a learned gate multiplies two branches. It gives better quality for the same compute than a plain GELU MLP.