Harnessing Domain Specialists in Multimodal Mixture-of-Experts for Efficient Adaptation

What it claims

Pretrained multimodal MoE models organize their experts semantically, even though nothing in training asks for it. Experts split by modality, image experts cluster by domain (medical, math, remote sensing), and visually different but related object classes share experts. ExpertLens reads this structure from the weights alone. It projects each expert's router weights onto vocabulary tokens to label what the expert handles, with no domain data needed. You then fine-tune only the experts for your target domain and freeze the other expert MLPs. Across sequential adaptation to math, medical and remote-sensing tasks, this matches or beats full fine-tuning while updating 21.7–47.0% of parameters, with a 4.0× average training speedup. It also beats LoRA on accuracy and trains 2.9× faster than LoRA.

How it works

• Models studied: Qwen3-VL-30B-A3B, Kimi-VL-A3B and Gemma-4 26B-A4B.
• Expert labelling: ExpertLens follows the Logit Lens idea. Router weights are decoded through the vocabulary to give tokens that name each expert's specialty.
• Domain selection: experts whose decoded tokens match the target domain are chosen without looking at any examples. Those experts are trained and the remaining expert MLPs are frozen.
• What the page omits: the excerpt does not give the exact selection rule, such as a threshold or top-k, or say how many experts are picked per domain.

Evidence

The structural findings behind the method:

• Modality split (Qwen3-VL-30B-A3B):
• Experts' text-activation share is bimodal: 462 experts sit at 0–5% text share and 433 at 60–65%.
• Hartigan dip test: dip = 0.0136, p < 0.001.
• Image specialization is more extreme than text, which spreads across 60–80%.
• Gemma-4 is similarly bimodal. Kimi-VL-A3B is closer to unimodal, which the authors attribute to its smaller expert pool.
• Domain structure (kNN purity, 12 balanced groups per grouping):
• Image-patch activations group best by dataset/domain, well above grouping by mean color or patch position.
• Text-token activations group best by part of speech, matching prior text-only MoE findings.
• This holds across layers in both Qwen3-VL and Kimi-VL.
• In t-SNE at layer 18, sample sets were 1.5K tokens/patches per dataset over six datasets, 18K activations in total.
• Object semantics (TinyImageNet):
• Setup: 200 classes, 19,900 class pairs, 100 images per class.
• Expert-set Jaccard overlap falls as WordNet Wu-Palmer distance grows (Pearson r = −0.380, p < 0.001).
• Related pairs such as sea slug/jellyfish and banana/lemon stay above the unrelated baseline, some at Z > 3.

The headline numbers (21.7–47.0% of parameters, 4.0× vs full FT, 2.9× vs LoRA) come from the abstract and introduction. The per-task results table is not in the fetched text.

How to use it

• Code: github.com/damianomarsili/ExpertLens
• Project page: glab-caltech.github.io/expertlens

Practical consequences for adoption:

• The method only applies to MoE models. On dense models it has no expert structure to exploit.
• Specialization is strongest on the vision side. Text experts organize by syntax more than by domain, so domain selection for text-heavy tasks may be weaker.
• On an MoE with a small expert pool like Kimi-VL-A3B, the separation may be less clean.

Limits

• The fetched text stops partway through §3, so the details of §3.4 (ExpertLens), §4 (adaptation experiments) and any limitations section are missing. That leaves out:
• per-benchmark scores and metrics
• the LoRA configuration (rank, target modules)
• hardware and compute
• which models were actually fine-tuned
• No licence for the code is stated on the page.
• "Matches or surpasses" full fine-tuning is a range claim. Check the per-task numbers in the paper before assuming parity in your domain.
• The evaluated domains are math, medical and remote sensing only.