Multi-View MoE: Letting Every Router Shape Expert Computation

What it claims

Multi-View MoE, from Baidu's ERNIE Team (arXiv, October 2026), changes the routing and expert-input interface of a LatentMoE backbone. It uses two sigmoid routers instead of one. Each selected expert also gets a router-weighted blend of two latent projections of the hidden state as its input. On a 24-layer 10B-A1B model trained on 708B tokens, the nine-task mean reaches 34.25, which is 1.18 points over the baseline. The motivation is a measured weakness of single routers. Their expert rankings agree poorly with dense expert-utility evaluation, and agreement is worst in middle and deep layers.

How it works

The backbone is LatentMoE (Elango et al., 2026). The router reads the full hidden state h. A shared down-projection P gives the latent input z = Ph, and each routed expert maps that latent to a latent. An always-active shared expert and an up-projection follow.

• Multi-Router. R routing heads (R = 2 by default) use distinct random seeds and have no diversity penalty. They score all N experts. Scores are summed after the sigmoid, because summing logits would collapse into one linear scorer. One top-k is then taken over the summed scores plus the backbone's balancing bias b. Output gates normalize the summed scores over the selected experts, with the bias excluded, and apply a per-expert learnable scale α.
• Multi-View. Each router is paired with one latent projection P_m, so K = R. View 1 reuses the backbone's P, and the other views are independently initialized perturbations of it. For each selected expert, its raw sigmoid scores are normalized across routers into proportions that sum to one. Those proportions form a convex mix of the views, and the mix is the expert's input. Mixing happens before dispatch, and each expert runs once.

Two experts can have the same summed score and still receive different inputs. Scores (0.4, 0.1) and (0.2, 0.3) both sum to 0.5 but give view proportions (0.8, 0.2) and (0.4, 0.6). No extra mixing network is added. The cost is K down-projections in place of one, plus extra router heads. Expert count, expert FLOPs and all-to-all volume do not change.

flowchart LR
  H[Hidden state h] --> R1[Router 1 sigmoid]
  H --> R2[Router 2 sigmoid]
  H --> V1[View 1 = P h]
  H --> V2[View 2 = P2 h]
  R1 --> S[Sum + bias b]
  R2 --> S
  S --> T[Top-k selection]
  R1 --> N[Normalize per expert across routers]
  R2 --> N
  T --> N
  N --> M[Mix views per expert]
  V1 --> M
  V2 --> M
  M --> E[Each expert runs once]

Evidence

• Downstream. At the 10B-A1B scale on 708B tokens, the full method scores 34.25 on the nine-task mean, +1.18 points over the baseline. Multi-Router alone gives +0.6.
• Routing alignment. Multi-Router raises the Spearman correlation between router score and expert utility by a relative 8.1% globally. In the layer with the weakest single-router alignment, the relative gain is about 58%. Utility is measured as each expert's output L2 norm, normalized by its mean over sampled tokens. Single-router layer-wise ρ stays below about 0.2 across the 24 layers.
• Mixing is used. At the final MoE layer, selection and gates were held fixed while view proportions were permuted across experts. This raised held-out NLL by 0.01340 nats/token. Permuting across tokens for the same expert also raised NLL.

Limits

• Results come from one model scale (10B-A1B, 24 layers) and one token budget.
• The fetched text stops partway through the method section. The per-task scores, the alternatives to post-sigmoid summation (Apps. A.3–A.5) and the ablations of learned mixing weights and gradient coupling (§4.5) are not visible.
• The utility metric is a proxy: output L2 norm, not task loss.
• The page names no code release, weights or licence.