Mixture of Experts

An ML technique that divides a model into multiple “expert” models.

Main components:

• The experts (the sub-learners/networks)
• A gating function

The output of the gate is used to combine the outputs of the experts in some way.

Given a learned gating network $G$ and the experts $E$, one typical example is:

$$y=\sum _{i=1}^{n}G(x)E_i(x)$$

Where the gating network with weights $W_g$ is defined as:

$$G(x)=\text{Softmax}(x\cdot W_g)$$

In plain English, the outputs of the experts are combined in a weighted sum weighted by the gating scores.

A more modern way of using MoE is to do sparsely-gated mixture of experts.