Mixture-of-Experts (MoE)

Mixture-of-Experts is the architecture trick behind almost every large model shipped since 2024: make the network enormous, but only wake up a small slice of it for each token. A dense model uses every parameter for every token, so doubling size doubles inference compute. An MoE model breaks that link. It can store the knowledge of a hundreds-of-billions-parameter model while doing the per-token work of a model a tenth that size. The idea dates back to Shazeer and colleagues in 2017, who showed a trainable gate could route inputs across thousands of sub-networks and scale capacity over 1,000x with only minor compute cost.

One layer, many experts, a router#

Inside a transformer block, most of the parameters sit in the feed-forward network (FFN), the part that runs after attention. An MoE layer replaces that single FFN with N parallel copies called experts, plus a tiny router network. For each token, the router scores all experts and sends the token to the top k, typically 1 or 2 out of anywhere from 8 to 256. Only those experts run; the rest are skipped entirely. The outputs of the chosen experts are blended by the router's scores and passed on. Attention layers stay dense and shared, so "experts" are not separate models, just alternative FFNs inside each layer.

Two refinements show up in modern designs. A shared expert runs for every token alongside the routed ones, holding common knowledge so the routed experts can specialize (DeepSeek and Llama 4 both do this). And routing happens per layer, per token: the same token can hit expert 3 in layer 10 and expert 41 in layer 11, so there is no clean "math expert" you could point at.

Total versus active parameters#

Every MoE model therefore has two sizes, and confusing them leads to bad capacity planning. Total parameters count every expert in every layer: the model's full store of knowledge, and what must sit in GPU memory. Active parameters count only what actually executes for one token: the attention layers, the router, and the k chosen experts. The numbers as of mid-2026:

• Mixtral 8x7B (the model that mainstreamed MoE in late 2023): 47B total, about 13B active, 2 of 8 experts per token.
• DeepSeek-V3 / R1: 671B total, 37B active, using many small fine-grained experts plus a shared one.
• Llama 4 Scout: 109B total, 17B active across 16 experts; Llama 4 Maverick: 400B total, the same 17B active, spread across 128 experts.
• Qwen3-235B-A22B: the name says it directly, 235B total with 22B active.
• Kimi K2: roughly 1 trillion total, 32B active.

Notice the pattern: total sizes have exploded toward a trillion while active sizes cluster between 13B and 40B. The "A22B" naming convention exists because active parameters, not total, predict what a model costs to run.

Why size decouples from inference cost#

Per-token compute in a transformer is roughly proportional to the parameters that participate in the forward pass. In a dense model that is all of them; in an MoE it is only the active set. Llama 4 Maverick makes the point cleanly: it carries 400B parameters of knowledge but each token pays for 17B, about the compute of a small dense model. Mixtral matched or beat Llama 2 70B on most benchmarks while doing less than a fifth of the per-token work. That is the whole bargain: capacity scales with experts, cost scales with the router's k.

The decoupling helps latency too. The decode phase of generation is memory-bandwidth-bound (see prefill vs decode): each step streams weights from memory. At small batch sizes only the active experts' weights need to be read per token, so an MoE generates tokens faster than a dense model of equal total size, not just cheaper.

What MoE costs you instead#

The savings are in compute, not memory. All 671B of DeepSeek-V3's parameters must be resident to serve it, because any token might route anywhere on the next step. Work the numbers: at 8-bit precision those weights alone are about 671 GB, so even before activations and cache you need a node of eight 96 GB GPUs just to hold a model that computes like a 37B. That pushes serving toward multi-GPU clusters with expert parallelism (different experts living on different GPUs), and it makes quantization more valuable: Meta notes Scout fits on a single H100 only after Int4 quantization, squeezing 109B parameters into roughly 55 GB. At high batch sizes the bandwidth advantage also fades, since a large enough batch touches every expert every step.

Training brings its own tax. Routers left alone collapse onto a few favorite experts, so training adds load-balancing objectives to keep all experts useful, and fine-tuning MoE models is touchier than fine-tuning dense ones. None of this is your problem if you consume models through an API, but it is exactly why self-hosting a trillion-parameter MoE is a cluster project, not a single-GPU one.

Practical takeaways#

Read model cards with two eyes: active parameters predict per-token speed and API price, total parameters predict memory footprint and (loosely) knowledge capacity. When sizing self-hosted hardware, budget VRAM for the total size plus KV cache, then estimate throughput from the active size. Do not assume a "17B-active" model behaves like a 17B dense model on knowledge-heavy tasks; it usually behaves much bigger. And expect the pattern to continue: sparse-total, small-active is now the default recipe for frontier models, because it is the only known way to keep growing what a model knows without growing what every token costs.