Mixture of experts (MoE)

Also written: MoE, experts

A model design that has many expert sub-networks and uses only a few per token. You need memory for all the parameters, but speed is closer to a model of the active size.