BeNice! Closing Looping's Inference-Memory Gap via KV-Sharing

What it claims

KV-shareability means reusing the final recurrence's KV-cache for every earlier recurrence without losing quality. In current looped language models it appears inconsistently, and the authors (UC San Diego and Together AI, October 2026) tie it to fixed-point convergence. Their recipe has two stages. Stochastic-depth pretraining makes the recurrent block contractive. BeNice! post-training then corrects one localized error that sharing introduces. They report that BeNice! raises shared-cache quality by up to 8.35 points on Core and 4.56 on Core-extended. Against standard finetuning, it improves shared-cache validation perplexity by 9.75 points while matching it when caches are unshared.

How it works

The models are middle-looped: a prelude block P, a recurrent block R applied T times with input injection, and a coda block C. The setup follows Parcae, with the spectral radius of the injection matrix A constrained below 1.

Training with a randomly sampled depth T (for example, Poisson-distributed) forces tied weights to give low-loss outputs at many depths. This drives residuals toward a fixed point. Because K and V projections are locally Lipschitz in the residual, the caches converge too. Swapping an intermediate cache for the final one then perturbs attention by an amount bounded by the residual gap. The final recurrence is empirically the best source cache to broadcast.

Contraction alone is not enough. In failing models, sharing produces a token-independent basis shift in the first layer of the first recurrence, and this moves the model to a different fixed point. BeNice! targets that shift. It injects the final recurrence's KV into the first recurrence and trains with an auxiliary cross-entropy loss on that shared path, alongside the standard loss on the unshared final recurrence. For MLA the shared object is the final recurrence's latent. For Kimi delta attention it is the final recurrence's state.

Evidence

• Existing models without BeNice! (Table 1, Lambada BPB unshared vs shared):
• Parcae 140M (0.962 vs 0.962) and 370M (0.762 vs 0.764) share cleanly.
• Parcae 770M (0.645 vs 1.460) and 1.3B (0.601 vs 2.281) break.
• Huginn 3.5B shares (0.429 vs 0.429; Core 40.97 vs 40.53).
• ACT-trained Ouro breaks badly: Ouro 1.4B Core falls from 54.57 to 25.17, and Ouro 2.6B from 59.78 to 26.58.
• Stochastic vs static depth (Parcae-style models trained from scratch, 140M to 1.3B): the shared-minus-unshared validation perplexity delta is between -0.02 and 0.02 with stochastic depth and between 1.31 and 3.10 with static depth. The Core delta is at most 0.25 with stochastic depth and up to 2.00 with static depth.
• Failure localization (validation perplexity, 770M / 1.3B):
• Unshared: 11.86 / 10.80.
• Full sharing: 34.24 / 1.71e4.
• Keeping only the first layer's first-recurrence KV: 11.86 / 10.80.
• Adding the mean projected error after layer 1 ("naive patching"): 11.87 / 10.84.
• Fixed-point agreement: the mean relative difference between shared and unshared final latents is at most 1.69% for models that share well and up to 177.5% for models that do not.
• Inference: in simulated long-context traffic, the cache-hit gain of up to 31.4% equals 8.1 absolute points.

How to use it

The page gives no code, weights or commands. The findings suggest a cheap check before adopting the method: evaluate an existing looped model with only the final recurrence's cache. Huginn 3.5B and Parcae 140M/370M already pass this check. Parcae 770M/1.3B and Ouro do not.

Limits

The fetched text stops at the start of Section 4. BeNice!'s exact objective weighting, training cost and the MLA/KDA and quantization results are not shown. The quantization formats are listed as fp8, mxfp8 and nvfp4 in the abstract but fp8, mxfp4 and nvfp4 in the introduction. The throughput and TTFT numbers come from simulation, not deployed serving. The authors report KV-sharing as a poor fit for ACT-style models such as Ouro and recommend stochastic fixed-point training instead.