Fault-tolerant foundation models

What it claims

Llama2-style transformers trained under the same random hardware faults they will see at inference ("fault-hardened") follow a modified scaling law. That law has an extra curvature term, α₂. Because of it, the fraction of the model doing useful work first falls with size and then recovers past a critical size N. Models trained clean and then run on faulty hardware ("fault-blind") have no well-defined scaling at all: past a fault-rate threshold, more training data makes them worse*. The authors read the recovery as evidence that models learn "good" error-correcting codes rather than repetition. Formal fault tolerance (finite capacity as N→∞ below a threshold fault rate p_th) is stated as a conjecture, not shown.

How it works

• Fault model. On every forward pass, in both training and inference, random blocks of k=4 matrix elements inside the attention and feed-forward matrix products are zeroed with probability p. Faults are resampled per query, so the model cannot calibrate to one fixed device defect.
• Capacity metric. η is the size of a fault-blind model, run on clean hardware, that matches the loss of a fault-hardened model of size N at fault rate p. The ratio η/N works like a code rate.
• Scaling law. L = E(p) + exp(α₀ − α₁n − α₂n²) + exp(β₀ − β₁d), with n = ln(N/N₀) and d = ln(D/D₀). N₀ and D₀ are both set to 4×10⁷, and the irreducible loss E is fixed at about 1.5. Standard laws have α₂ = 0. Here α₂ grows with p, which makes ln(η/N) a parabola in ln N with its minimum at N* = N₀·exp(c₁/2c₂).
• Logit diagnostics. The KL divergence between clean and faulted outputs is split into two parts:
• a temperature shift, which rescaling the logits can fix;
• an irreducible "scrambling" term, which it cannot.

Evidence

• Compute and data. About 40,000 GPU-hours, spent on thousands of models trained on a slice of FineWeb across a sweep of N, D and p. The largest models are N ≈ 10⁹.
• How α₁ and α₂ move with p. At small p, drops in α₁ are offset by rises in α₂. At large p, α₂ plateaus while α₁ keeps falling. The authors read this as a sign of a fault-tolerance threshold.
• Training cost. β₁ falls as p rises, so fault-hardened models need more tokens to match a fault-free model's data budget. The size of this overhead grows with p (Eq. 6, Fig. 2c), partly by extrapolation.
• Fault-blind collapse (Fig. 3a). With faults on at evaluation, loss can increase with D. The D at which gains stop appears to fall as p_eval rises. Standard errors come from 8 independently trained models per D.
• Temperature. Fault-blind models' output temperature rises quickly with p_eval, and more so at larger D. Fault-hardened models hold a constant temperature until p_eval exceeds p_train.
• Scrambling. Fault-hardened models lose almost no capacity to the irreducible scrambling term while p_eval < p_train, independent of D. Fault-blind models degrade from p_eval > 0. Logit rescaling alone therefore cannot rescue a clean-trained model.
• Robustness of the trend. The authors say the capacity-recovery trend shows up in the processed raw data without the fitted law (Supplementary Information Sec. A).

Limits

• Simulated faults only. Results come from emulated block-zeroing faults, not real hardware. The page describes no physical device test. Other fault types are not shown, such as arbitrary-magnitude errors or analog noise, even though the intro names those as the realistic case.
• Asymptote not resolved. The term A that governs behaviour as N→∞ is indistinguishable from 0 at the tested scales, and L_N/E never dropped below about 0.5. Whether capacity plateaus at a nonzero constant at commercial scale is untested. The Fig. 1 extrapolations depend on unmeasured assumptions.
• Fixed constants. E was fixed by hand because it could not be identified from the data. The authors say this does not change the qualitative results. k was fixed at 4 throughout.
• Matched fault rates. Hardening assumes p_train = p_eval. Protection holds only up to p_train.
• Evaluation scope. Only held-out loss on web text is reported. There are no downstream task benchmarks.
• No release or energy numbers. No energy-savings figures, code, weights or licence appear in the fetched text.
• Truncated page. The fetched text cuts off before the hardware architecture section and Methods.