Common-Mode Collapse and Recovery in Direct Feedback Alignment

What it claims

When Direct Feedback Alignment (DFA) uses tanh hidden units, independent sigmoid outputs with BCE, and plain SGD, training stalls for several hundred steps near the loss of a constant class-frequency predictor, with accuracy near chance. The cause is the common mode of the output error: the mean error ē that is shared across inputs. At initialization, predictions near 0.5 against one-hot targets make ē large. Fixed random feedback sends ē to each hidden unit with a different random coefficient, which drives tanh units into saturation ("representation collapse").

The fixes that work are:

• initialising the readout bias to the class prior
• subtracting the batch mean of the error
• centering inputs and freezing hidden biases together

Replacing errors with their signs keeps the collapse going. Class information mostly stays decodable through the collapse. The cost is learning speed.

How it works

• Exact decomposition. The DFA update ΔW = −η⟨δhᵀ⟩ splits into a covariance term plus a rank-one term −η δ̄ h̄ᵀ.
• At initialization, each unit's broadcast mean error (Bē)ᵢ is distributed N(0, g²‖ē‖²).
• This shifts each unit's preactivation mean by dμ/dt = −η sech²(μ)(Bē)ᵢ(H+1). The +1 comes from the bias update.
• Why it persists. With BP or pre-aligned feedback (B ∝ W_outᵀ), hidden drift lowers the mean-only loss. Random feedback gives no systematic correction on average, so only the readout learning the prior shuts the drive off.
• Collapse number κ. κ is a dimensionless estimate built from g‖ē₀‖, the activity energies, and η_out/η.
• It does not change when both learning rates are scaled together.
• At large dose, gate participation scales roughly as (15/8)√(2/π)/κ.
• Reduced model. A Gaussian-quadrature model is initialized from the network with no fitted parameters. It predicts plateau loss, onset, and the layer ordering of participation minima.

Evidence

Baseline setup. MNIST, a 3-layer tanh MLP of width 300, Xavier-uniform weights, zero biases, g=1, SGD with η=η_out=10⁻³, batch size 128. Diagnostics use a 2,048-example probe set. Results average 15 DFA runs and 5 matched-BP runs.

During the plateau:

• The minimum per-unit variance across inputs falls to 0.028 of its initial value, against 0.83 for BP.
• The shared-error norm drops from 1.278 to 0.10 within 100 updates.
• The input-dependent error RMS stays at 0.90–0.95 throughout.
• The top-layer DFA gradient is slightly anti-aligned with the BP gradient.
• Raw DFA gradient estimates are several times larger than BP's.

Model fit. The reduced model was tested across 48 settings × 3 seed pairs.

Plateau scaling. The fitted plateau length is T ∝ (η_hid·g)^−0.40 · η_out^−0.60. Higher gain deepens collapse but shortens the plateau.

Optimizers and other settings:

• Adam learns faster than SGD despite deeper collapse.
• Related effects appear in deeper networks, convolutional networks, and CIFAR-10. Severity depends on the readout, optimizer, and input statistics.

How to use it

If you train DFA (or similar broadcast-feedback rules) and see loss parked at the class-prior value:

1. Initialise the output bias to the class prior.
2. Or subtract the minibatch mean of the error before projecting it.
3. Or centre the inputs and freeze the hidden biases. The paper finds that only both together prevent collapse.

Weakening the feedback gain trades less collapse for slower learning. Avoid sign-of-error variants in this setting.

The fetched text gives no code, repository, or per-intervention step counts.

Limits

• Setting-specific. The stall is tied to sigmoid/BCE readouts with tanh units and SGD. In the authors' reconstruction of Nøkland's original protocol, the stall lasts only tens of updates. At the larger tested learning rate, BP also saturates.
• Where the closure fails. The closure describes the onset, not the plateau. The ignored residual is 0.01 of the leading term at initialization but exceeds it by step 300.
• What the model misses. It does not reproduce participation rebound or unit turnover during recovery. It predicts almost no drive for two-class and squared-error settings, yet collapse is still measured there.
• Scope of the mitigation result. Batch-mean error subtraction is reported as helping "in the tested setting" only.
• Truncated source. The fetched text cuts off before the detailed intervention, architecture, and CIFAR-10 results, so their magnitudes are not given here.