Sliding Networks: A Family of Residual Structures Preserving Feature Learning and Feature Diversity

What it claims

Datong Zhou (Sorbonne Université) and Blake Bordelon (UT Austin) posted this preprint on 24 October 2026. It argues that the common shape choice N♯ ≍ N♭ forces N♭/(N♯L) → 0 as depth grows. That yields an ODE limit in which random depth variation, and with it feature diversity, washes out. Keeping the ratio at a fixed α > 0 gives an SDE limit that keeps both diversity and locally maximal feature learning. On CIFAR-10 and Tiny Shakespeare, the authors estimate parameter savings of 20% for the SDE recipe and 40% for Half Copy. The comparison is against the ODE recipe's mean training loss after fixed training schedules (Appendix H.4).

How it works

The block is a pre-attention-free residual MLP. The residual update is scaled by β0·√N♭/(L·N♯), with standard Gaussian weights rescaled per Table 1:

• Initialization std: input 1, up 1/√N♭, down √N♭/K, output 1/N♭, where K = N♯L.
• Learning rates: for Adam, input 1, up 1/N♭, down 1/K, output 1/N♭. For SGD, input N♭, up K/N♭, down N♭/K, output 1/N♭.
• Muon: has its own learning-rate and spectral-clipping scalings. Its limit also requires N♯ → ∞.

For SGD and Muon, residual variables stay jointly Gaussian through depth and training. Adam does not generally preserve this.

Diffusion rate. At initialization, the residual variance grows with c = β0²α. For ReLU without normalization, C(τ) = C(0)·e^(cτ/2). With unit pre-RMSNorm, it is C(0) + cτ/2.

Raising β0 is not a fix. Increasing β0 to compensate for small α keeps the initialization variance, but the first-step energy change grows as (β0·η_eff)². Stability then requires η_eff = O(1/β0), which pushes hidden-feature learning toward a "semi-freeze." Correlated initialization (Half Copy) is the alternative the authors propose for conventional shapes.

Sliding. Sliding regroups layers at fixed K. Gluing two layers stacks their up projections and concatenates their down projections. This doubles N♯ and halves L, and the operation is reversible.

Evidence

• One-step theory check. The one-step variance formula matches finite-network means at L = 128 and β0 = 1, across α values of 1/2, 1 and 4/3.
• Finite-size error. Predicted RMS error is O(1/√N♭ + 1/L + 1/√(N♯L)) for SGD and Adam. For Muon it is O(1/√N♭ + 1/L + 1/N♯).
• Convergence. On CIFAR-10 residual MLPs with N♭ = q² and N♯ = L = q, output and kernel errors fall with q, consistent with O(1/q).
• GPT-2. Measurements show randomly initialized residual blocks can dominate the input embedding at initialization. This places practical models on the SDE-like side rather than the ODE side.
• Shape at matched budget. Among the tested Tiny Shakespeare shapes, sliding everything into one wide block gave the lowest training loss.

Limits

• Analysis setting. The analysis starts from MLP blocks without biases or normalization. Attention is handled by treating each head as one channel (Appendix B.5).
• Training loss only. The reported savings are conservative estimates on training loss over fixed schedules. Generalization results are not reported in the fetched text.
• Depth unresolved. The limit says only K = N♯L matters, yet language models perform differently under regrouping at fixed parameter count. Converging to a limit therefore does not settle how to split width and depth.
• Partial text. The fetched text ends at the start of the experiments section. Half Copy's construction, the full results tables and the Muon scalings were not available in readable form.