Mixture of experts
A mixture-of-experts layer holds many feed-forward networks but sends each token to only one or two of them.
Many experts, few used
In an MoE layer the single feed-forward network is replaced by several experts and a small router. For each token the router scores the experts and picks the best. Only the chosen expert runs, so the layer is sparsely activated: total parameters can grow a lot without the cost per token growing with them.
Capacity
Experts run in parallel, so each has a fixed capacity: (tokens ÷ experts) × capacity factor. A token whose chosen expert is already full is dropped from that layer. A larger factor drops fewer tokens but wastes compute on empty slots.
In the lab, six tokens are routed to three experts. Change the capacity factor and re-roll the router scores.
A worked example
Six tokens, three experts, capacity factor 1: each expert takes at most 6 ÷ 3 × 1 = 2 tokens. If three tokens choose expert 1, the third is dropped and skips the layer (its residual connection carries it through unchanged). With a factor of 1.5 each expert takes 3 tokens and nothing is dropped, but on average a third of the slots stay empty and their compute is wasted.
Total versus active parameters
A model with 8 experts per layer and top-2 routing stores 8 feed-forward networks per layer but runs only 2 for each token. Its total parameter count is several times that of a dense model with the same cost per token. The memory still has to hold every expert, so MoE models save compute, not memory.
Easy to confuse
- Experts are not topic specialists by design; any specialisation comes from training and is often hard to name.
- Routing is per token and per layer, so one sentence can use many different experts.
What is the main benefit of a mixture-of-experts layer?
Each token only runs through a few experts, so the compute per token stays near that of a small model while the model as a whole holds far more parameters.
6 tokens, 3 experts, capacity factor 1. How many tokens can each expert take?
Capacity = (tokens ÷ experts) × C = (6 ÷ 3) × 1 = 2. If three tokens choose the same expert, one of them is dropped.
What does the router do?
The router is a small learned network. Its scores decide which expert or experts process each token.
Read more in the books
- AI Engineering, Chip Huyen: Ch. 2, the section on model size (mixture-of-experts).
- TinyML and Efficient Deep Learning Computing, Lecture 12: Transformer and LLM (slides), Song Han, MIT 6.5940: section IV, mixture of experts.