Mixture of experts

Also written: MoE

A design where each block has many MLPs and a small router sends each token to only a few. The model holds many parameters but uses only some per token.