RailWave: Adaptive Spatial and Temporal Scheduling for Expert-Parallel Communication

What it claims

RailWave argues that in Rail-optimized GPU clusters, expert-parallel MoE All-to-All traffic suffers from two problems that routing and expert placement do not fix. The first is hot source Rails. The second is receiver incast. RailWave addresses both below the routing layer and leaves the demand matrix and each token's destination expert unchanged. Two findings from the authors' own analysis motivate it:

• Expert-load balance does not predict Rail balance. Across 900 GLM-4.5-Air training step–layer pairs, a linear fit from expert-load imbalance gives R² = 0.019 for source-Rail imbalance and R² = 0.296 for destination-Rail imbalance. Six cases with nearly identical expert-load max/mean (8.2–8.4) had busiest destination-Rail shares ranging from 15.5% to 24.1%.
• Transfer ordering alone changes latency. With the same rank-to-rank demand, payload, backend and number of rounds, raising maximum receiver fan-in from 2 to 3 increased summed per-layer P50 dispatch–combine latency from 1.801 s to 3.009 s, a 67.1% increase.

How it works

• RailBalance. For each node pair, records are split into quotas across the eligible Rails, with quotas differing by at most one record. Remainders rotate deterministically across destination slices. Surplus records are matched to deficit slots by sorted order, which avoids atomic slot allocation. Only records that move need relocation, and the saved owner-to-proxy map lets combine restore original token positions. The method assumes a "Rail-symmetric" topology, meaning egress Rail r maps to ingress Rail r. Per-source balance is guaranteed; aggregate balance is not.
• Permutation scheduling. In wave k, node s sends only to node (s+k) mod N, giving N−1 waves. Each node has at most one sender and one receiver per wave. Peer plans depend only on group membership and are built once. This contrasts with FAST, whose demand-dependent Birkhoff schedules must be rebuilt as routing changes.
• Four paths and a selector. The paths are Native, Rail-only, Permutation-only and Joint (RailBalance applied within each wave). Each phase is described by three features: average remote payload per GPU (B), global source-Rail skew (ρ_g) and receiver concentration (κ).

flowchart TD
  A[Phase traffic: B, ρg, κ] --> Z{Remote payload?}
  Z -- no --> N[Native]
  Z -- yes --> L{Within ε of a calibrated anchor?}
  L -- yes --> P[Nearest anchor's path]
  L -- no --> J[Joint]
  O[Offline profiling] --> T[Anchor table] --> L

An offline anchor keeps a path other than Joint only when repeated measurements are stable and show a conservative advantage over Joint. All ranks follow the same path and wave order for a given dispatch–combine pair.

Evidence

The evaluation used two 32-GPU clusters, one H800 and one H20, running communication workloads replayed from real GLM-4.5-Air (106B) training. RailWave reached up to 5.84× (H800) and 4.36× (H20) communication speedup over Native DeepEP. The fetched text stops before the evaluation section, so the per-path breakdowns, typical (non-peak) speedups and end-to-end training impact are not available here.

How to use it

The code is at https://github.com/CyberSecurityErial/RailWave-EP. It is built on DeepEP. Adopting it requires an offline profiling pass on your own cluster to build the anchor table; the normalization and retention criteria are in the paper's Appendix A.3. The page gives no install commands or licence.

Limits

• The headline numbers are "up to" figures, measure communication only, and come from only 32 GPUs on two clusters.
• Permutation scheduling serializes N−1 waves and keeps the original imbalance in node-pair demand. It pays off only when the reduction in incast outweighs the extra rounds, which is why the selector exists.
• RailBalance gains depend on Rail skew versus relocation cost, and its guarantees assume Rail-symmetric topology.
• Selector decisions are valid only near calibrated operating points. Anything else falls back to Joint, so recalibration is likely needed for new hardware or models.
• The paper is a v1 preprint posted 2 Oct 2026.