Telescopic Language Models

What it claims

A nested Transformer cascade, the unchanged Matryoshka Language Model Suite (MLMS) architecture, has 20 layers and exits at blocks 11, 16 and 20 (50M/100M/200M). Trained with MLMS's fixed-exit objective, every depth except the trained exits is unusable: validation perplexity is about 20 at the exits but 10²–10⁵ (up to about 4×10⁵) elsewhere. Retraining the same cascade with stochastic prefix supervision plus a full anchor makes all 20 prefixes valid language models. Perplexity falls smoothly from 81 at k=1 to 15 at k=20. The authors conclude the off-exit collapse comes from the objective, not from the nesting.

How it works

• Cascade. Llama-style sub-models of strictly increasing width. Each consumes the previous sub-model's output, concatenated with a fresh slice of the shared input embedding. Each has its own final RMSNorm and LM head. The k-prefix is the first k blocks plus the norm and head of the sub-model that contains block k.
• Per-step loss. L = λ·ℓ(F_k̃(x), y) + γ·ℓ(F_N(x), y), with k̃ resampled from π every step. Defaults are λ = γ = 1 and π uniform over depths 1..N. Both passes use the same micro-batch and the same next-token target. There is no teacher and no reduced target.
• Uneven updates. Block j gets the anchor gradient every step but the prefix gradient only when k̃ ≥ j. Shallow blocks therefore update most often.
• Optional distillation. A distillation term from the stop-gradient full model is ablated. The method does not need it.
• Inference. Truncate at any k and run. No extra parameters or routing.

Three samplers are studied:

• Uniform: the only one that trains every prefix, including k=1.
• Log-uniform: k = ⌈N^u⌉, roughly p(k) ≈ 1/k. The mean k is about 6.9, and k=1 is never sampled.
• Exit-grid: concentrated on MLMS's exits.

Evidence

The setting is a 200M proxy suite on 20B FineWeb-Edu tokens, with an identical data stream for all methods.

• Whole-curve quality (AULB, lower is better). TLM scores 3.28 against 5.73–5.90 for the MLMS suites. AULB is mean token NLL integrated over k = 1..N. The gap is 30–44% under FLOP-weighted, median and deployment-range variants.
• Below the smallest exit. TLM serves ten operating points below MLMS's 50M exit, down to 0.11× full latency at PPL 26–81. MLMS's own prefixes collapse there.
• Full capacity.
• Uniform TLM reaches 14.99 PPL; the MLMS suites span 14.96–15.42.
• A separately trained vanilla 200M model reaches 13.98, a 7.1% gap.
• The exit-grid sampler reaches 14.65 on a single seed. That is a 2.2% edge over MLMS (within seed noise) and narrows the vanilla gap to 4.8%, but it gives up off-exit coverage.
• Collapse mechanism. A readout-rescue control attributes most of the MLMS collapse to off-exit heads that never saw those prefixes. Repairing only the head still leaves a prefix about 6× behind one trained end-to-end.
• Training cost.
• TLM (uniform): 131 GPU-hours.
• Distillation-free MLMS suite: 149 GPU-hours.
• Vanilla 50M/100M/200M trio: 216 GPU-hours.

TLM delivers 20 operating points; the other two deliver 3. Per-step cost stays at two forward-backward passes however many depths are covered, whereas MLMS's cost grows with the number of exits.

• Deployment metric (LODA). A throughput-weighted frontier area that counts only valid operating points. The paper notes this is by construction a coverage metric.

How to use it

The fetched text gives no code, weights, licence or commands. To reproduce, you need:

• the MLMS cascade architecture;
• Eq. 1 with λ = γ = 1;
• a sampler for k̃, where uniform gives a continuum and an exit-concentrated sampler gives better quality at chosen depths;
• a budget of two forward-backward passes per step.

Choosing the sampler density π decides at training time which depths you favour. The cascade itself stays the same.

Limits

• Scale. Only a 200M-parameter, 20B-token proxy is tested. Nothing larger is reported.
• Full-size quality. The model trails an independently trained twin (14.99 vs. 13.98 PPL). The authors call this the price of sharing the trunk.
• No free lunch from the sampler. No sampler wins on both coverage and per-exit quality. In per-exit comparisons the MLMS suite keeps an edge, reported in §4.2. The 14.65 result is a single seed and within seed noise.
• Metric choice. AULB weights each unit of depth equally, which magnifies the off-exit explosion. The authors treat the per-prefix curves as the primary evidence.
• Shallow prefixes are weak. Even TLM's shallowest prefixes are poor, at PPL 81 for k=1.
• Truncated page. The fetched text stops partway through the cost accounting. Ablation tables (no-anchor, LayerDrop, distillation) and downstream task numbers are referenced but not shown.