Rethinking Routing Exploration in Modern Mixture-of-Experts

What it claims

Adding shared experts to a fine-grained MoE can cancel the benefit of activating more experts. In matched pretraining runs from 100M to 2.4B total parameters, MoEs with shared experts stop beating no-shared-expert MoEs as the activated-expert count K grows. At 498M and 2.4B they get worse as K rises. MirrorMoE fixes this by suppressing unneeded routed experts per token, derived by "mirroring the routing logits via sign negation". On the 2.4B model, the authors report gains of 0.67 points in pretraining and 1.64 points downstream.

How it works

The diagnosis comes before the method. All diagnostics use a 498M-total / 121M-active model with E=66, K=8, and S=2 shared experts, checked at 20%, 50% and 100% of training. Each metric is lower with shared experts than without:

• Expert ablation. Zeroing a shared expert raises perplexity far more than zeroing any routed expert.
• Routing criticality. Swapping in uniform random routing one layer at a time costs much less perplexity in the early and middle layers. Those layers barely depend on the learned router.
• ExpertVar. This measures the relative output change when one routed expert is swapped in, averaged over all routed experts. It is lower in every layer. The authors prove that R·ExpertVar upper-bounds the output change from any routing, assuming renormalized gate weights.
• RouterVar. This is the output variance when expert sets are sampled from the router distribution, with 256 samples per token. It drops by about an order of magnitude. The same trend holds with Gumbel-perturbed routing at β ∈ {0.5, 2}.

The argument is that shared experts dominate the output, so different routes produce similar representations and the router cannot tell routes apart. Meanwhile, going from (E,K)=(8,1) to (64,8) expands the number of possible routes from 8 to about 4.4×10⁹.

The paper uses unnormalized gates (Z=1, as in DeepSeek-MoE and Qwen1.5-MoE) and deterministic top-(K−S) routing.

Evidence

Figure 1 reports validation perplexity (lower is better) with fixed E, R ∈ {64,128} and S=2, at K = 6 / 8 / 10. The values below are reconstructed from extracted chart labels in legend order.

• Shared experts lose their edge as K grows. At K=10, the no-shared baseline beats the shared-expert baseline at every scale.
• MirrorMoE is best in every cell.
• MirrorMoE mostly restores gains from larger K. At 498M, 625M and 2.4B, its perplexity falls steadily as K rises.

Limits

• Method details are missing. The fetched text ends partway through Section 2.3. The exact MirrorMoE formulation, its compute overhead, its theoretical adaptivity result and the downstream benchmarks behind "1.64 points" are not in the text.
• The 100M result runs the wrong way. MirrorMoE perplexity rises with K (51.58 → 52.07), so "restores gains from larger K" does not hold there. The 100M model still beats both baselines at every K.
• Scale is modest. The largest model tested is 2.4B total parameters, far below the production MoEs the paper motivates with (DeepSeek, Kimi, Qwen).
• Gate normalization may matter. The variance bound assumes renormalized gates, but the experiments use Z=1. Check which applies to your architecture.
• Table values need confirming. They come from chart label extraction and should be checked against the PDF figure.
• No code link. The abstract says code is available, but the link did not survive extraction.