LLM Lab

Mixture of experts

A mixture-of-experts layer holds many feed-forward networks but sends each token to only one or two of them.

Where this sits in a transformer
Tokenizer→Embedding→Positional encoding→Multi-head attention→Feed-forward→LayerNorm + residual→Linear head→Sampling

Many experts, few used

In an MoE layer the single feed-forward network is replaced by several experts and a small router. For each token the router scores the experts and picks the best. Only the chosen expert runs, so the layer is sparsely activated: total parameters can grow a lot without the cost per token growing with them.

the0.70.20.1→ expert 1
cat0.10.80.1→ expert 2
sat0.20.30.5→ expert 3
The router gives each token a score per expert (three experts here). Each token goes to its top expert, outlined, so only one expert’s weights run for that token.

Capacity

Experts run in parallel, so each has a fixed capacity: (tokens ÷ experts) × capacity factor. A token whose chosen expert is already full is dropped from that layer. A larger factor drops fewer tokens but wastes compute on empty slots.

In the lab, six tokens are routed to three experts. Change the capacity factor and re-roll the router scores.

A worked example

Six tokens, three experts, capacity factor 1: each expert takes at most 6 ÷ 3 × 1 = 2 tokens. If three tokens choose expert 1, the third is dropped and skips the layer (its residual connection carries it through unchanged). With a factor of 1.5 each expert takes 3 tokens and nothing is dropped, but on average a third of the slots stay empty and their compute is wasted.

Total versus active parameters

A model with 8 experts per layer and top-2 routing stores 8 feed-forward networks per layer but runs only 2 for each token. Its total parameter count is several times that of a dense model with the same cost per token. The memory still has to hold every expert, so MoE models save compute, not memory.

Easy to confuse

Check yourself · 1 of 3

What is the main benefit of a mixture-of-experts layer?

Check yourself · 2 of 3

6 tokens, 3 experts, capacity factor 1. How many tokens can each expert take?

Check yourself · 3 of 3

What does the router do?

Read more in the books

  • AI Engineering, Chip Huyen: Ch. 2, the section on model size (mixture-of-experts).
  • TinyML and Efficient Deep Learning Computing, Lecture 12: Transformer and LLM (slides), Song Han, MIT 6.5940: section IV, mixture of experts.
All books
Labroute the tokens
How to read this
  • The router table gives each token (row) a score for each of the three experts (columns); the outlined cell is the expert it picks.
  • Each expert column below holds a fixed number of slots: (tokens ÷ experts) × capacity factor. Tokens that arrive when their expert is full are dropped.
  • Drag the capacity factor and re-roll the scores to see when tokens are dropped.