Learning What to Share: Expert Specialization for Time-Series Forecasting

What it claims

The paper argues that mixture-of-experts gains in multivariate forecasting come from informative channel allocation, not just from added capacity. On frozen PatchTST and iTransformer, a fixed per-channel expert mixture ("Persistent ID") beats input-dependent routing on average. DLinear is the exception: it favors input conditioning. PCR builds on this finding. It is a residual add-on to a frozen forecaster, and its PCR-K4 variant cuts dataset-balanced MSE by 3.86-14.25% against the three frozen backbones. It beats expert-budget-matched MoLE, CCM and VE on average, with fewer trainable parameters and lower median single-window latency.

How it works

Each channel's forecast gets a correction: y_c = f(X)_c + α·a_c(X)·Σ_e π_e·A_e·z_c(X).

• z_c is taken from the frozen backbone's prediction-head features.
• Each expert A_e is a low-rank map from those features to the H-step horizon.
• The weights π are nonnegative and sum to one.
• α is one global scale, chosen on validation.
• a_c undoes window normalization. For DLinear, a_c = 1.

PCR trains by alternating two steps. With the experts fixed, each channel's π_c is solved exactly as a constrained ridge least-squares problem, with at most K nonzero weights. With π fixed, the experts take gradient steps. The weights are "persistent": they stay the same across input windows. The backbone stays frozen throughout.

flowchart LR
  B[Frozen backbone] --> Z[Head features z_c]
  Z --> S[Solve π_c: constrained ridge LS]
  S --> T[Train experts by gradient]
  T -->|repeat| S
  T --> O[Corrected forecast]

Evidence

The controlled comparisons use 11 datasets: ETTh1/h2/m1/m2, Electricity, Traffic, Weather and PEMS03/04/07/08. The setup is L=96 and H∈{96,720}, with three seeds and TSLib backbones. The expert bank has four rank-16 experts, two active per channel. The Global correction uses a single rank-64 map to match the budget.

Table 1 reports MSE reduction over the frozen backbone (DLinear / PatchTST / iTransformer):

Persistent ID beats Random routing on 9/11 datasets for DLinear and on 10/11 for both transformers. With all four experts active ("Dense ID"), the reductions are 3.76%, 13.40% and 9.79%, so sparsity is not required.

Persistent ID's lead over Identity + input routing is small: 2.51% on PatchTST (9/11 wins) and 0.36% on iTransformer (6/11 wins).

Unfreezing the backbone and fitting it jointly with the experts lowers MSE further, by 0.72%, 1.71% and 3.83%. For the transformers, this costs 2.53-2.75× the fitting time and 2.85-7.35× the peak training memory. DLinear fits faster jointly.

GalaxyTS is the synthetic benchmark. It has 512 channels and 12,000 steps per realization, with three realizations at each separation level s∈{0, 0.33, 0.66, 1}. Each channel mixes four echo components that differ only in delay. Exact optimal shared and channel-specific MSEs are known, so the benefit of specialization can be measured directly. In matched corrections on GalaxyTS, persistent mixtures and ridge grouping both beat random and input-dependent routing. Persistent mixtures gain more over Global correction.

Limits

• Training and evaluation code is not public. The authors say a release is planned.
• Errors are reported in standardized coordinates.
• GalaxyTS's exact references cover only predictors that use one channel's history.
• In GalaxyTS, the parameter s changes family separation and echo strength together, and the authors note the sweep does not isolate either factor.
• On DLinear, persistent routing loses clearly to input-conditioned routing. The authors attribute this to the added nonlinearity.
• The fetched text stops partway through the GalaxyTS results. The per-dataset PCR tables, latency figures and parameter counts were not available.