Predictable Scaling: Efficient Model Exploration via Scaling-Law Prediction

What it claims

Predictable Scaling (PS), from Meta (arXiv v1, October 2026), predicts the relative normalized-entropy (NE) gap between a candidate change and a baseline recommendation model. It does this from one scaled-down proxy run instead of a full training run. The authors' central observation is that relative gaps between hypotheses behave as smooth, low-dimensional functions of scale even when absolute curves are noisy. Among "actionable" hypotheses (effect larger than the A/A noise floor τ_AA), PS preserves rank perfectly (Spearman ρ = 1.0) at 79 to 88% compute reduction.

How it works

The target is Δ_i = (ℓ_i − ℓ_0)/ℓ_0 on NE, where a negative value means improvement. Proxies are built from six scaling functions, each set by a ratio α in (0, 1]. They compose multiplicatively:

• DD: hash-seeded data subsampling (FLOPs scale with α)
• HSR: fewer embedding table rows, minimum 5K (memory scales with α)
• ES: narrower embedding width (FLOPs and memory scale with α)
• FS: feature pruning by importance score (FLOPs scale with α)
• WAS: all hidden dimensions scaled uniformly (FLOPs and memory scale with α²)
• DS: truncation to ⌊αL⌋ layers (FLOPs scale with α)

Three predictors map proxy gaps to full-scale gaps:

• Raw proxy: the proxy gap is used as the prediction.
• OLS linear regression: fitted on the proxy gaps.
• Empirical Gaussian Process (EGP): its prior comes from historical NE training curves. The mean is the pointwise average and the covariance is the empirical sample covariance. It extrapolates the proxy's NE over training steps to convergence.

Proxy selection enumerates compositions of up to three functions that satisfy (N+1)·r·C_full ≤ B. It trains them on a benchmark corpus of hypotheses with known full-scale gaps and picks the configuration that minimizes MAE under cross-validation. PS depends on four stated conditions: convergence stability, gap transferability, sign preservation, and low-dimensional structure.

flowchart TD
  A[Hypothesis + baseline] --> B[Proxy run via scaling profile]
  B --> C[EGP extrapolates NE curve]
  C --> D{CV above threshold?}
  D -- no --> E[Use predicted gap]
  D -- yes --> F[Run full-scale training]

The EGP rejects any prediction whose coefficient of variation exceeds a threshold. A rejected hypothesis goes to full-scale training.

Evidence

All results use M = 1 proxy, and MAE is reported in units of τ_AA.

TS-1 is a task-specific ranker with 3,403 MFLOPs per forward pass, 112 trainers and 80B samples on H100s. Results on its CTR head, over 14 held-out hypotheses:

All three predictors give ρ = 1.0 on the 6 actionable hypotheses. The scaling profile was DD 0.317, HSR 0.5 and ES 0.25. It cut trainers by 86% and samples by 80%.

FM-1 is a foundation ranker with 117,734 MFLOPs per forward pass, 1,024 trainers and 122B samples. Results over 7 held-out hypotheses:

So on this larger model the raw proxy alone exceeds the noise floor on both CTR heads. The FM-1 profile was DD 0.5, HSR 0.2, ES 0.2, FS 0.2 and DS 0.5. It gave 79% compute reduction, with 88% fewer trainers and 50% fewer samples.

Design trade-off: WAS reduces the NE gap by 38% relative to ES at α = 0.5.

The paper cites a public reproduction on Criteo with four public models in Appendix H, but that appendix is not in the fetched text.

How to use it

Adopting PS requires three things:

• a benchmark corpus of hypotheses with known full-scale gaps, to select the proxy;
• historical NE training curves, to build the EGP prior;
• a measured A/A noise floor τ_AA.

The fetched text gives no code, package or licence.

Limits

• Small held-out sets. There are 14 and 7 hypotheses, and the standard errors are about as large as the means. The authors call the 15% EGP gain "indicative."
• Untested settings. Only M = 1 and offline NE were evaluated. Multi-scale proxies and online metrics are listed as future work.
• Sign reversals. Compositions with sign-reversal rates above 15 to 20% should be rejected.
• Unverified architectures. For architectures where the four conditions are unverified, the authors recommend PS screening combined with selective full-scale validation. DCN-v2 results motivate architecture-aware choice of scaling ratios.
• Data access. The industrial datasets are not public.
• Truncated text. The fetched text ends partway through the appendix.