Mixture-of-Experts (MoE)

Mixture of experts (MoE) is a model architecture in which each layer holds many sub-networks, called experts, and a small router sends each token to only a few of them. A dense model uses every parameter for every token, so doubling its size doubles the work per token. An MoE model breaks that link: it can hold hundreds of billions of parameters while doing the per-token work of a model a tenth that size.

The idea is older than large language models. Shazeer and colleagues showed in 2017 that a trainable gate could route inputs across up to thousands of feed-forward sub-networks and raise model capacity more than 1,000 times with only minor losses in computational efficiency. Most of the large open-weight models released since 2024 use it. The closed frontier labs do not publish their architectures, so nobody outside them can say the same for their models.

One layer, many experts, a router#

Inside a transformer block, most parameters sit in the feed-forward network, the part that runs after attention. An MoE layer replaces that single network with many parallel copies, the experts, plus a router. For each token the router scores every expert, sends the token to the top few, and blends their outputs by those scores. The others are skipped. Attention stays dense and shared, so an expert is not a separate model, only an alternative feed-forward block inside one layer.

The numbers vary widely. Mixtral 8x7B sends each token to 2 of 8 experts. DeepSeek-V3 sends it to 8 of 256 routed experts plus 1 shared expert, and Kimi K2 to 8 of 384 plus 1 shared. The shared expert runs for every token and holds common patterns so the routed ones can specialise; DeepSeek and Llama 4 both use one. Routing also happens per layer and per token, so the same token can hit one expert in layer 10 and a different one in layer 11. There is no "maths expert" you could point at, and that matters if anyone proposes pruning experts for a narrow use case.

Total versus active parameters#

Every MoE model has two sizes, and confusing them leads to bad capacity plans. Total parameters count every expert in every layer, and all of them must sit in GPU memory. Active parameters count what runs for one token: attention, the router and the chosen experts. As of October 2026:

Totals have passed a trillion while active counts run from about 5B to about 50B. The "A22B" naming exists because the active count, not the total, predicts what a model costs to run per token.

Why size decouples from compute#

Per-token compute in a transformer is roughly proportional to the parameters that take part in the forward pass. In a dense model that is all of them, and in an MoE it is the active set. Llama 4 Maverick holds 400B parameters, and each token pays for 17B. Mixtral's paper reports that it outperforms or matches Llama 2 70B and GPT-3.5 on every benchmark it evaluated, with 13B active, under a fifth of the dense model's per-token work. Compare models on per-token cost with the active count, and never use it to guess what a model knows.

Speed follows at small batch sizes. Generating each token is limited by how fast weights stream from memory (see prefill vs decode), and an MoE only has to read the chosen experts' weights for each token. A large enough batch touches every expert on every step, though, and the advantage over a dense model of the same total size fades. If you plan to serve at high concurrency, measure throughput at your batch size rather than reading it off the active count.

What MoE costs you instead#

The savings are in compute, not memory. All 671B of DeepSeek-V3's parameters must be resident, because any token can route anywhere on the next step. At 8 bits per weight, the precision DeepSeek released it in, the weights alone are about 671 GB. That does not fit a node of eight 80 GB H100s, which holds 640 GB, before any room for the KV cache. A node of eight 141 GB H200s fits it with room to spare. So a model that computes like a 37B dense model is served like a cluster job, usually with experts spread across GPUs (expert parallelism, covered under model parallelism).

Check the quantized size before you price hardware, because quantization is often what brings an MoE model onto fewer GPUs. Meta says Llama 4 Scout fits on a single H100 only with Int4 quantization, which brings 109B parameters to about 55 GB. OpenAI ships gpt-oss-120b with its expert weights in a 4-bit format so that it runs on one 80 GB GPU.

Training adds its own costs. A router left alone collapses onto a few favourite experts, so training adds load-balancing objectives to keep all of them in use, and fine-tuning an MoE model is touchier than fine-tuning a dense one.

Read both numbers on the model card#

For anyone calling a model through an API, MoE shows up only in the price and the speed, and both largely follow the active count. The total count matters the day you consider self-hosting. Bigcircle recommends sizing a self-hosted MoE in this order: memory from the total parameters at the precision you will run, plus the KV cache for your context lengths and concurrency; then throughput from the active parameters; then an eval on your own task. The last step is not optional, because a 17B-active model does not behave like a 17B dense model on knowledge-heavy work. It usually behaves much larger, and it may still miss on your domain.