Skip to content

Back to the catalog

Data Scarcity and Model Sparsity: Mixtures-of-Experts Overfit More to Repeated Data
arxiv.orgpaper

Data Scarcity and Model Sparsity: Mixtures-of-Experts Overfit More to Repeated Data

This paper studies how Mixture-of-Experts (MoE) language models respond to repeated training data compared to dense Transformers, across models from 80M to 1B active (8.5B total) parameters. It finds MoEs overfit to repeated data much faster than dense models — degrading noticeably at 4x repetition versus 8x for dense models — with the effect scaling with total (not active) parameter sparsity, and shows that dropout and masking-based regularization can partially mitigate this while MoE routing stabilization and expert specialization correlate with the overfitting.

TL;DR MoE models overfit to repeated training data far more severely than dense Transformers, with degradation scaling with total sparsity.
Open original ↗
Document

What it claims

Beyond the summary's "MoEs overfit faster," the abstract puts specific crossover points on the degradation. In all-unique data settings MoEs beat dense models, but under repetition they lose that edge: MoEs begin to suffer at 4x repetition (vs. 8x for 80M dense), deteriorate rapidly, and after 32x repetition actually underperform dense models. The effect scales with total, not active, parameters. Regularization pushes the crossover back rather than closing the gap: with strong masking-based regularization, MoEs outperform dense models even past 64x repetition — but the authors state no method fully matches the performance of training on all-unique data.

How it works

The study varies data repetition rates across two axes the summary does not name: single- vs. multi-domain data mixes, and MoE configuration by expert count and granularity. Models span 80M–1B active (8.5B total) parameters. For the mechanism analysis, they report that MoE routing "universally stabilizes early in training," and that the degree of expert specialization correlates with how badly a model overfits repeated data — the paper frames over-specialization as the memorization pathway to disrupt.

Evidence

The comparison is against densely activated Transformers as baseline at matched scales. Concrete thresholds: 80M dense repeats 8x with minimal degradation; MoEs degrade from 4x, cross below dense at 32x, and (with masking regularization) stay ahead past 64x. Dropout is named as a partial remedy. These are the abstract's stated figures; the abstract does not give the loss/perplexity metric behind "degrade" or the exact regularization masking scheme.

Limits

This is an abstract only — no tables, metric definitions, or per-configuration numbers are on the page, so the exact expert counts, granularities, and domain mixes tested are unstated here. The central caveat is the authors' own: no regularization method fully recovers all-unique-data performance, so masking-based fixes buy repetition headroom, not a free lunch. The mechanism claims (routing stabilization, expert specialization) are stated as correlations, and the paper positions memorization-disruption as future work rather than a delivered method.

Technologies named here

  • Mixture-of-Experts
  • Transformer

Techniques named here

  • Mixture of experts
  • Data repetition
  • Dropout
  • Masking-based regularization

Related in research