Glossary
What is a mixture of experts (MoE)?
In short
A mixture of experts (MoE) is a neural network design in which only a small part of the model works on each word. A component called the router picks that part.
Language models · mixture of experts
MMOOZZGG.com editorsChecked: 1 min read
For a child
At school there is a math teacher, a music teacher and an art teacher. When you have a question about notes, you go to the music teacher, not to all of them at once. A helper at the door tells you whom to ask. Some neural networks work the same way: there are many teachers inside, and only the two right ones answer each question.
In plain words
An ordinary neural network uses all its parts for every word. A mixture-of-experts model has many separate blocks inside, called experts, and a router. The router looks at the next word and sends it to a few suitable blocks, while the others rest. The model holds a huge amount of knowledge, but each step costs the computing of a small model. Think of a hospital: it has many doctors, but only the right specialists examine a patient, so the visit is fast.
For business
This design gives the quality of a large neural network at a lower computing cost; open models such as Mixtral use it. The catch is memory: the resting experts must still sit in graphics card memory, or the model cannot answer. So you need a server sized for the full model, while the answer speed is close to a small model’s. The cost of ownership depends on the total number of parameters, the load and how many users work at the same time. Before you buy, check how much memory the whole model takes after compression and whether the server can handle peak load. It helps to compare server configurations for different models in advance.
For a specialist
A mixture of experts (MoE) replaces a dense layer of a neural network with a set of parallel expert sub-networks and a trained router (gating network). For each token, the router picks k experts out of N and weighs their outputs. This is conditional computation: the number of parameters grows, while the cost per token stays almost the same. Shazeer et al. (2017) showed a sparse layer with thousands of experts and up to 137 billion parameters in total between LSTM layers. In transformers, the experts usually replace the feed-forward block. Mixtral 8x7B (Jiang et al., 2024) has 8 experts in each layer, and the router picks 2 per token: 47 billion parameters are available, 13 billion are active. Engineering issues include load balancing between experts through an auxiliary loss, a cap on expert capacity, and expert parallelism in distributed training.
Example
A company chooses an open model for its own server. A mixture-of-experts model answers as fast as a small one, but it needs as much graphics card memory as all its experts together, and that decides which server to buy.
Related terms
Questions and answers
Sources
Not sure whether your company needs this?
AI solutions map