Learning Length-Extrapolatable Recurrent Models

What it claims

Beyond what the summary states, the abstract makes a specific empirical argument for why gradient decay is not the culprit: dense per-token losses can still train a shared recurrent rule "despite severe decay," so decay alone does not determine whether learning fails. This is the observation that motivates reframing the problem around state credit rather than gradient magnitude.

How it works

Two details the summary does not carry. First, the rescaling is direction-preserving: CST "locally rescales the state-credit signal to stabilize its norm without rotating the component being corrected" — it changes magnitude, not direction, of the credit signal, and only in the backward pass. Second, CST is not a single method: because "controlled synthetic tasks and real data exhibit different credit dynamics," the authors specialize CST separately to each regime. So adopting this means picking (or deriving) a regime-appropriate variant rather than applying one uniform rule.

Evidence

The abstract reports gains beyond the training horizon "at up to 128x the training length" on both synthetic and real data. No baseline numbers, metrics, task names, model sizes, or compute are given on this page — only the extrapolation multiple and the two-regime split.

Limits

This is an abstract-only arXiv page (submitted 8 Sep 2026); there are no tables, hyperparameters, baselines, or ablations to inspect. Key things a reader must check in the PDF before relying on it: what the "different credit dynamics" between synthetic and real data actually are and how the two CST specializations differ concretely; what tasks and models the 128x claim covers; and whether the direction-preserving rescale interacts with the parameter update in a way that needs tuning per architecture. The claim that decay does not determine failure is asserted here, not demonstrated on the page.