Accordion: Elastic-Depth Inference via Fixed-Prefix Layer Looping

What it claims

Accordion-6B-A0.5B is one set of MoE weights that runs at five fixed depths, from 24 to 36 layer executions. The extra depth comes from running some layers a second time. The paper shows the shared model costs almost nothing against separately trained models at each depth: in 80B-token controlled runs, it lands within 0.003 nats of validation loss of budget-specific specialists at the three budgets tested, including the unlooped path.

At full scale (3.6T tokens, 8K context), deeper inference improves 36 of 37 benchmark scores. The middle budget, High, captures 89% of the 3.01-point gain. High uses 6 extra layer executions against Ultra's 12. Released checkpoints are Accordion-Base, Accordion-Base-32K, Accordion-Instruct and Accordion-RL, all trained on public datasets.

How it works

The base is a 24-layer pre-norm MoE with these settings:

• top-8 routing with a sigmoid router
• query–key layer norm
• a reduced DeepSeek-V3 tokenizer with 128,000 entries
• untied embeddings

Budget k repeats the first k layers of the window 7–18, with k ∈ {0, 3, 6, 9, 12}. The budgets are named Low, Medium, High, XHigh and Ultra. Each selected layer runs twice in a row, not as a multi-layer block or a whole stack.

The second pass shares attention, experts, norms and the router with the first. It is scaled by per-channel tanh gates that start at zero, so at initialization the second pass is an identity map. The gates add only 2md parameters. Expert-load balancing biases are kept separately for the first and second passes, in the style of DeepSeek-V3's auxiliary-loss-free method. A small sequence-level balance loss (λ = 1e-4) is also used.

flowchart TD
  A[Request picks budget k] --> B[Layer ℓ first pass]
  B --> C{ℓ in first k of layers 7-18?}
  C -- no --> E[Next layer]
  C -- yes --> D[Same weights again, tanh-gated, own KV cache]
  D --> E

Training samples one budget uniformly per micro-batch, from a shared seed, so MoE all-to-all stays aligned across workers. There is no cross-budget consistency loss. SFT keeps training all five budgets. RL trains three domain experts (instruction following, math, code), each covering all budgets. These are merged by parameter-group merging, then refined with on-policy distillation from the teacher that matches the student's domain and budget.

Evidence

The design ablations use a 1.61B MoE trained on 80B tokens of Nemotron-CC-v2. Results are held-out validation loss in nats.

• Layer selection: eight strategies were compared at equal expected depth (30 executions), four fixed and four randomized. A fixed prefix over layers 7–18 was lowest at all four looped budgets. Its mean advantage was 0.0025–0.0041 nats.
• Number of budgets: training five budgets beat training three or four at the same expected compute.

• Router: a shared router (mean 2.5724) beat a separate second-pass router (2.5737) and a separate router that excludes first-pass experts (2.5741).
• Post-training: Accordion-RL beats each domain expert on that expert's task-and-budget average. It outperforms four public reference models on IFEval; the excerpt does not name them. Extra depth still helps math and code after RL.

How to use it

The budget is chosen once per request and fixed for every token, with a static execution graph per budget. FLOPs scale with L+k layer executions. The second pass keeps its own KV cache, so KV memory is 1.25× at High (k=6) and 1.5× at Ultra (k=12), relative to Low. Accordion-Base-32K adds 50B tokens of long-context training. The excerpt gives no licence, install command or measured latency.

Limits

Each design configuration was trained once, and most loss gaps are a few thousandths of a nat. The specialist comparison covers only three of the five budgets. Depth is fixed per request, not adaptive per token or per input. The best budget differs across tasks after post-training. Measured runtime depends on communication and kernel efficiency, and only analytical costs are given. The fetched text stops partway through §2.2.4, so the budget-conditioned parameter results and the full benchmark tables are not covered here.