MMOOZZGG.com

Glossary

What is a mixture of experts (MoE)?

In short

A mixture of experts (MoE) is a neural network design in which only a small part of the model works on each word. A component called the router picks that part.

Language models · mixture of experts

MMOOZZGG.com editorsChecked: 1 min read

For a child

At school there is a math teacher, a music teacher and an art teacher. When you have a question about notes, you go to the music teacher, not to all of them at once. A helper at the door tells you whom to ask. Some neural networks work the same way: there are many teachers inside, and only the two right ones answer each question.

In plain words

An ordinary neural network uses all its parts for every word. A mixture-of-experts model has many separate blocks inside, called experts, and a router. The router looks at the next word and sends it to a few suitable blocks, while the others rest. The model holds a huge amount of knowledge, but each step costs the computing of a small model. Think of a hospital: it has many doctors, but only the right specialists examine a patient, so the visit is fast.

For business

This design gives the quality of a large neural network at a lower computing cost; open models such as Mixtral use it. The catch is memory: the resting experts must still sit in graphics card memory, or the model cannot answer. So you need a server sized for the full model, while the answer speed is close to a small model’s. The cost of ownership depends on the total number of parameters, the load and how many users work at the same time. Before you buy, check how much memory the whole model takes after compression and whether the server can handle peak load. It helps to compare server configurations for different models in advance.

For a specialist

A mixture of experts (MoE) replaces a dense layer of a neural network with a set of parallel expert sub-networks and a trained router (gating network). For each token, the router picks k experts out of N and weighs their outputs. This is conditional computation: the number of parameters grows, while the cost per token stays almost the same. Shazeer et al. (2017) showed a sparse layer with thousands of experts and up to 137 billion parameters in total between LSTM layers. In transformers, the experts usually replace the feed-forward block. Mixtral 8x7B (Jiang et al., 2024) has 8 experts in each layer, and the router picks 2 per token: 47 billion parameters are available, 13 billion are active. Engineering issues include load balancing between experts through an auxiliary loss, a cap on expert capacity, and expert parallelism in distributed training.

Example

A company chooses an open model for its own server. A mixture-of-experts model answers as fast as a small one, but it needs as much graphics card memory as all its experts together, and that decides which server to buy.

Related terms

Questions and answers

Why is a mixture-of-experts model faster than a regular model of the same size? At each step, only the chosen experts work, not the whole network. So the model needs about the same computing per word as a model several times smaller.
Does an MoE model need less memory? No, it needs memory for its full size. All experts must be loaded, because the router may choose any of them for the next word.

Sources

Not sure whether your company needs this?

AI solutions map
Contact us